Opened 8 years ago
Closed 8 years ago
#16299 closed enhancement (fixed)
upgrade numpy/scipy to 1.8.1 and 0.14
Reported by:  fbissey  Owned by:  

Priority:  major  Milestone:  sage6.3 
Component:  packages: standard  Keywords:  
Cc:  Merged in:  
Authors:  François Bissey  Reviewers:  Nathan Dunfield 
Report Upstream:  N/A  Work issues:  
Branch:  5a16124 (Commits, GitHub, GitLab)  Commit:  5a161248dd34be17b790e79a00414354abf0311f 
Dependencies:  Stopgaps: 
Description (last modified by )
Upgrade numpy/scipy to latest versions. scipy 0.14 has finally ditched oldnumeric which means it won't generate warnings all over the place. A few doctests will have to be updated.
Package sources:
Change History (18)
comment:1 Changed 8 years ago by
 Description modified (diff)
comment:2 Changed 8 years ago by
 Branch set to numpy181
comment:3 Changed 8 years ago by
 Created changed from 05/07/14 01:42:37 to 05/07/14 01:42:37
 Description modified (diff)
 Modified changed from 05/08/14 11:05:48 to 05/08/14 11:05:48
comment:4 Changed 8 years ago by
 Branch changed from numpy181 to u/fbissey/numpy181
 Commit set to 62c44eee3c74b71ce7a08357d68c8fcb707ed0ae
comment:5 Changed 8 years ago by
 Status changed from new to needs_review
Woopsie. Anyway I have updated the checksums and SPKG.txt for numpy and scipy. No changes where necessary to the packages themselves as far as I could tell. I fixed three doctests:
 src/sage/functions/exp_integral.py: the last digit of a test changed, I put "..." since it is probably a little unstable.
 src/sage/numerical/optimize.py: also a numerical precision thing. In this case we ask for a value within 0.0001 tolerance but tests the answer to 16 decimal places. No surprise when we have changes behind the 5th place.
 src/sage/modules/vector_double_dense.pyx: a doctest warning has changed code.
comment:6 Changed 8 years ago by
comment:7 Changed 8 years ago by
 Status changed from needs_review to positive_review
I tried out this branch and the new packages with Sage 6.2 on two systems: OX Mavericks and RHEL 6.3 (gcc 4.4). Both numpy and scipy installed cleanly and the three modified doctests passed on both systems. Other changes in the patch look fine, so setting status to "positive review".
comment:8 Changed 8 years ago by
 Reviewers set to Nathan Dunfield
comment:9 Changed 8 years ago by
 Status changed from positive_review to needs_work
Breaks doctests on OSX (10.6 and 10.9):
sage t long src/sage/tests/french_book/calculus_doctest.py ********************************************************************** File "src/sage/tests/french_book/calculus_doctest.py", line 216, in sage.tests.french_book.calculus_doctest Failed example: find_root(expr, 0.1, pi) Expected: 2.0943951023931957 Got: 2.0943951023931953
comment:10 Changed 8 years ago by
OK, will amend that. Any other failures I missed?
comment:11 Changed 8 years ago by
 Commit changed from 62c44eee3c74b71ce7a08357d68c8fcb707ed0ae to 5a161248dd34be17b790e79a00414354abf0311f
Branch pushed to git repo; I updated commit sha1. New commits:
5a16124  Fix doctest failing on OS X on the last digit after upgarding numpy/scipy to 1.8.1/0.14.0

comment:12 Changed 8 years ago by
 Status changed from needs_work to needs_review
OK fixed the doctest and pushed. Ready for another round of review.
comment:13 Changed 8 years ago by
I don't see any reason why this should give a different result on OSX. Do you?
comment:14 Changed 8 years ago by
Not sure. We are talking about a difference in the 4E16 which is in the relative tolerance of the function and well within its own tolerance.
def find_root(f, a, b, xtol=10e13, rtol=4.5e16, maxiter=100, full_output=False):
But I would have been more ready to accept a difference based on CPU arch rather than OS on similar CPU.
Looking at scipy the underlying c code hasn't changed since 2008 but higher up tolerance checking has been enforced in 0.14rc2 (scipy/optimize/zeros.py) that's the only thing I can see having an impact.
comment:15 followup: ↓ 16 Changed 8 years ago by
 Status changed from needs_review to positive_review
I did a little testing, and definitely the issue is a result of the scipy upgrade and not the numpy one. It also shows up on OS X version 10.8 as well.
My experience is that the last digit in numerics like this can be remarkably sensitive to the OS/CPU/compiler version/phase of the moon. I once had two Ubuntu 12.04 virtual machines running on the same physical hardware where the last digit differed like this; one had a 32bit kernel and the other a 64bit one, which you wouldn't think would make a difference...
I think this sort of thing is to be expected, and it certainly seems harmless enough in this instance. So I've changed the status back to "positive review".
comment:16 in reply to: ↑ 15 Changed 8 years ago by
Replying to dunfield:
I did a little testing, and definitely the issue is a result of the scipy upgrade and not the numpy one. It also shows up on OS X version 10.8 as well.
My experience is that the last digit in numerics like this can be remarkably sensitive to the OS/CPU/compiler version/phase of the moon. I once had two Ubuntu 12.04 virtual machines running on the same physical hardware where the last digit differed like this; one had a 32bit kernel and the other a 64bit one, which you wouldn't think would make a difference...
Oh, yes I would. But then I have seen tickets where that happens in the last 6 years.
I think this sort of thing is to be expected, and it certainly seems harmless enough in this instance. So I've changed the status back to "positive review".
It is technically harmless it is however a curiosity because the change is sudden. If I had time I would check other versions of scipy to narrow down when the change happened and try to understand why.
comment:17 Changed 8 years ago by
In the end I spent the time checking https://github.com/scipy/scipy/commit/8edf03864b7c3e43e652ac154f98e96ee4913687 and https://github.com/scipy/scipy/commit/0ee24117ad63399b7a3353269de4e916fe409ad7 are responsible. I didn't try to pin it on one or the other. So it looks like we approch the root slightly differently between OS X and linux. My guess is that without the enforcement of tolerance OS X would go through one (or) more iteration(s) and end up with the same value as Linux.
comment:18 Changed 8 years ago by
 Branch changed from u/fbissey/numpy181 to 5a161248dd34be17b790e79a00414354abf0311f
 Resolution set to fixed
 Status changed from positive_review to closed
OK so it