Jekyll2019-07-31T08:23:23+00:00http://www.tensorworld.org//feed.xmlTensor WorldBy tensor nerds, for tensor loversJeremy CohenA few new tensor toolboxes2019-07-31T00:00:00+00:002019-07-31T00:00:00+00:00http://www.tensorworld.org//discussions/2019/07/31/some-new-interesting-toolboxes<p>There has been a few additions to <a href="/toolboxes.html">the list of tensor toolboxes</a>: Hottbox, TensorToolbox.jl. Here is a quick words about these interesting softwares.</p>
<!--more-->
<h2 id="tensortooboxjl-julia">TensorToobox.jl (julia)</h2>
<p>For thoses who want to start working with the now quite mature <a href="https://julialang.org/">Julia</a> programming language, there has been recently <a href="https://github.com/lanaperisa/TensorToolbox.jl">a new toolbox</a> released in Daniel Kressner’s team, that mimicks the well-known tensor toolbox on matlab. It has the particularity to handle tensor train formats.</p>
<p>Care however that Julia is a fast changing language, so keep updated for various changes in the syntax.</p>
<h2 id="hottbox-python">Hottbox (python)</h2>
<p>A team of researchers at the Imperial College of London have recently been
publishing several papers related to classification with tensor data and
models. They have now released their own toolbox in Python, <a href="https://github.com/hottbox/hottbox">HOTTBOX</a>, which
interestingly contains several tools for getting started easily with such
classification problems.</p>Jeremy CohenThere has been a few additions to the list of tensor toolboxes: Hottbox, TensorToolbox.jl. Here is a quick words about these interesting softwares.Changes to the website2019-07-30T00:00:00+00:002019-07-30T00:00:00+00:00http://www.tensorworld.org//news/2019/07/30/changes-to-the-website<p>It has been roughly one year since the last news on this website. Here are some
of the changes that have implemented recently, in the hope to center the scope
of tensorly towards useful information for everyone.</p>
<h2 id="simplying-the-website">Simplying the website</h2>
<p>As you will see, there are fewer sections, and fewer text. I have kept the
important content for now: the list of toolboxes (which has been updated, check
next post).</p>
<h2 id="adding-a-list-of-todo">Adding a list of todo</h2>
<p>There is a lot to do to make tensorworld a truly useful place for researchers
to find information about tensors and tensor research. I ahve listed a few
priorities in “The project” section, go and have a look if you might be able to
contribute.</p>
<!--more-->Jeremy CohenIt has been roughly one year since the last news on this website. Here are some of the changes that have implemented recently, in the hope to center the scope of tensorly towards useful information for everyone.TensorLy: a first step towards vertical collaboration?2018-09-30T00:00:00+00:002018-09-30T00:00:00+00:00http://www.tensorworld.org//discussions/2018/09/30/tensorly-starting-point<p>Since the TRICAP2018 conference, where it was decided that the signal
processing community should try to work together to produce, among others, an efficient
software for tensor problems (see the presentation
<a href="https://gitlab.com/cohenjer/tensor_world/blob/master/Presentation_file/pres_pdf.pdf">here</a>),
nothing has really been done to actually start the development of such a
toolbox. This is probably due to the immense work that is required to lay the
first stones to this project.
In this post, I would like to draw the attention of our community on a newly
developped package for Python:
<a href="http://tensorly.org/stable/index.html">TensorLy</a>. I will first present the
package, then discuss its advantages and flaws with respect to the other toolboxes available
for MATLAB such as Tensorlab or Tensor Toolbox, and finally I will try to
provide a guide map for how to improve TensorLy to match our goals.</p>
<!--more-->
<p>Note: If any statement in the following discussion appears to be even partially
incorrect, please mention it asap in the comments below, or email me at
tensor(dot)world(at)gmail(dot)com.</p>
<h1 id="tensorly-a-front-end-library-for-python">TensorLy, a front-end library for Python</h1>
<p>Although this was new to me before I started doing some research on tensor
packages this summer (2018), it appears the question of lacking tensor decompositions
software on free platforms such as Python has been around for quite some time.</p>
<p>In particular, <a href="https://www.oreilly.com/ideas/lets-build-open-source-tensor-libraries-for-data-science">this
article</a>
from 2015 already states partially the problem of lacking efficient tensor
software, especially for the machine learning community, in spite of obvious
interest in such tools. Anima Anandkumar, who was interviewed at the time, and
has been promoting the use of tensors in machine learning for quite some time,
recently associated with Jean Kossaifi and Yannis Panagakis to try and fill
this gap.</p>
<p>TensorLy is the result of this process. It is a Python library built upon
widely used libraries, in particular <a href="http://www.numpy.org/">NumPy</a> and
<a href="https://pytorch.org/">PyTorch</a>. This means that, when using the TensorLy
syntax to implement tensor factorizations and manipulations, one does not need
to worry about the way basic operations are handled. In particular, using
PyTorch as a back end, it is possible to make use of GPUs to speed up
computations in a completely transparent manner to the user.</p>
<p>TensorLy is relatively recent project, thus only a few functions of interest
for signal processing people are implemented, namely PARAFAC, Tucker and their
nonnegative counterparts.</p>
<h1 id="what-is-missing-for-tensorly">What is missing for TensorLy</h1>
<p>Before stating what I think is a very important chunk of content missing from
TensorLy, I want to point out that it does a good job, in its current version,
at handling tensor operations such as vectorizations, matricizations, tensor
computation from the Kruskal form, Kronecker products and so on. This is
because TensorLy relies on other efficient implementations of very basic tensor
operations (provided for instance by PyTorch).
To illustrate how TensorLy works, here is a shamelessly taken image from the
official publication <a href="https://arxiv.org/pdf/1610.09555.pdf">link</a>:</p>
<p><img src="/assets/TensorLy_Structure.png" alt="Stolen image" /></p>
<p>However, for those familiar with tensor decomposition problems, we know that
the set of mathematical tools required is quite large. In short, tensor
decompositions are based on, at least, four ingredients:</p>
<ul>
<li>data visualization, in order to asses the type of tensor model to apply, to
estimate the rank, to remove corrupted data, and so on.</li>
<li>data manipulation, which was discussed above.</li>
<li>optimization, to efficiently identify the parameters of decomposition models.
This is a crucial point, that has been extensively discussed in the tensor
literature and has known quite important improvements over the years.</li>
<li>modeling, which means being able to specify complex structures on the tensor
data, as well as the tensor decomposition parameters. The champion toolbox
for flexible modeling is the MATLAB <a href="https://www.tensorlab.net/">tensorlab</a> v3.0 and further, see the
docs for many detailed examples.</li>
</ul>
<p>Sadly, TensorLy falls somewhat short in anything related to visualisation,
optimization and modeling. To illustrate this, I have used the example of
Nonnegative CPD/PARAFAC (which I know quite well, thus the choice). See this
<a href="http://nbviewer.jupyter.org/url/www.tensorworld.org/assets/tensorly_nncpd_v2.ipynb">Jupyter Notebook</a> for a tentative comparison of the native TensorLy
solution with the numpy nonnegative least squares solver used alternatively.</p>
<h1 id="we-should-help-each-other">We should help each other!</h1>
<p>These missing pieces in TensorLy are exactly what us, signal processing
researchers, are good at. Personally, I think trying to start from scratch to
make yet another tensor toolbox on Python would be quite stupid, since
libraries such as TensorLy and PyTorch already offer a lot of flexibility.</p>
<p>Therefore, I think the (curently void) toolbox part of the TensorWorld project should merge
with the TensorLy project, to incorporate the previously mentioned features. In particular,
I myself am very sensitive to the optimization issue. Right now, TensorLy
implements ALS PARAFAC and Tucker, and multiplicative update (ugh) for
nonnegative PARAFAC. It
is possible to use methods inherited from the back-ends, such as BFGS and
Stochastic gradient. However, I would love the following to be easily doable
in TensorLy:</p>
<ul>
<li>Using <a href="http://www.cvxpy.org/index.html">CVXPY</a>, which makes optimization easy to write and is
efficiently implemented. CVXPY is based on conic optimization which is
extremely flexible and powerful. <a href="https://arxiv.org/abs/1506.04209">Recent works</a> have shown that proximal
methods used in an alternating fashion perform very well for a wide variety
of tensor decomposition problems. Therefore, CVXPY can be used
straightforwardly on the
per-mode derivation of most decomposition problems, see the example notebook.</li>
<li>Using personal code for specific problems, such as nonnegative least
squares. For instance, I like the A-HALS method for solving NNLS. Supposing
this method is implemented in a separate module, I would like to be able to
call it to perform the NNLS steps in Nonnegative PARAFAC.</li>
</ul>
<p>I am not a Python expert nor a hardcore programmer, but at the very least,
using CVXPY as a back-end seems reasonable. In any case, we need more
constrained decompositions available. Also, multiplicative update is not a
great algorithm in most scenarios, see for instance <a href="https://arxiv.org/pdf/1401.5226.pdf">this article</a> for the matrix
case.</p>
<h1 id="to-conclude">To conclude</h1>
<ul>
<li>TensorLy is a cool project, but could be much cooler with more people
contributing (a.k.a. this community).</li>
<li>It is a starting point that we can build upon, not just a pile of words with
no actions following up. This makes our life easier.</li>
<li>Contributing is accessible to anyone who knows Python, and the developers
have a detailed description on <a href="http://tensorly.org/stable/development_guide/index.html">how to contribute</a>.</li>
</ul>
<p>Let me know what you think of all this in the comments below! Also, please take
some time (2 minutes) to complete the <a href="/survey/">survey</a>
on this website!</p>
<p>EDIT (12/10/2018): after actually testing CVXPY for tensor decompositions, I am starting to think that maybe implementing methods such as NNLS, or AODAMM for a more general framework, would be very beneficial. It does not change the fact that optimization is one key point we could start working on alongside the Tensorly People.</p>Jeremy CohenSince the TRICAP2018 conference, where it was decided that the signal processing community should try to work together to produce, among others, an efficient software for tensor problems (see the presentation here), nothing has really been done to actually start the development of such a toolbox. This is probably due to the immense work that is required to lay the first stones to this project. In this post, I would like to draw the attention of our community on a newly developped package for Python: TensorLy. I will first present the package, then discuss its advantages and flaws with respect to the other toolboxes available for MATLAB such as Tensorlab or Tensor Toolbox, and finally I will try to provide a guide map for how to improve TensorLy to match our goals.My take on why we need Tensor_World2018-08-07T00:00:00+00:002018-08-07T00:00:00+00:00http://www.tensorworld.org//discussions/2018/08/07/reproducible-research<p>Tensor_World is my way to start a movement for more open and reproducible
algorithmic research in the tensor decomposition community. As of now, every
research group has their own toolbox, and they are disparate in their contents
and use. Therefore, it can be cumbersome to compare one’s new method with
existing ones because so many different software should be handled, with
various inputs, outputs and specifications.</p>
<!--more-->
<p>On this website, a <a href="/pages/toolboxes/">table</a> of all the existing toolboxes in various languages
that can compute any tensor decomposition model is being built in order for
everyone to naviguate this software jungle. As one can see, we are going in a
direction where concertation is mandatory, otherwise we will keep implementing
the same functions over and over again, while older well-implemented softwares
are forgotten.</p>
<p>In the end, what I am aiming for is easy reproducibility. Things should be as
simple as this: write you code in
whatever language, convert it to a package for Tensor_World (and benefit from well-implemented routines - one may think about the CPD/PARAFAC), and compare with acknowledge methods all implemented by
their authors as packages to the core libraries of Tensor_World.</p>
<p>I may be aiming for too much, but this is what I think we should be going for.
Along the way, I am setting a few tools for enabling fluid and easy
collaborations among us (Gitlab group repository, Google groups mailing list,
this webpage, see links in the nav bar). Feel free to join me in this project
by adding content anywhere you see fit, posting on this website what you think
of this project, or just come talk to me in the next conference.</p>Jeremy CohenTensor_World is my way to start a movement for more open and reproducible algorithmic research in the tensor decomposition community. As of now, every research group has their own toolbox, and they are disparate in their contents and use. Therefore, it can be cumbersome to compare one’s new method with existing ones because so many different software should be handled, with various inputs, outputs and specifications.The Tensor_World website is up and running2018-08-07T00:00:00+00:002018-08-07T00:00:00+00:00http://www.tensorworld.org//news/2018/08/07/website-is-up<p>Yeah!</p>
<p>Will come shortly: Comments on news and discussions through Disqus</p>Jeremy CohenYeah!