Updated README.md, INSTALL, and setup.py for v3.0 release

INSTALL

@@ -15,10 +15,9 @@ Basic steps to install:
 
   If you are on a modern Intel processor and have a recent version
   of GCC, you can get much improved performance by adding:
-  --enable-sse2 --enable-avx
+  --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-fma
 
-  Note that Ubuntu has good FFTW packages now (libfftw3-bin and
-  libfftw3-dev).
+  Ubuntu has good FFTW packages: libfftw3-bin and libfftw3-dev
 
 2.  Install PGPLOT.  http://www.astro.caltech.edu/~tjp/pgplot/
   You need the X-windows and postscript drivers at a minimum.
@@ -29,7 +28,7 @@ Basic steps to install:
   environment variable specified to allow the programs to link and to
   allow PGPLOT to find its important files.
 
-  Note that Ubuntu has good PGPLOT packages now (pgplot5).
+  Ubuntu has a good PGPLOT package: pgplot5
 
 3.  Install TEMPO.  Make sure to set the TEMPO environment variable.
   http://tempo.sourceforge.net/
@@ -50,8 +49,7 @@ Basic steps to install:
   a pulsar data standard that external tools can actually
   view!  How about that!)
 
-  Once again, Ubuntu has CFITSIO packages (libcfitsio3 and
-  libcfitsio3-dev)!
+  Ubuntu has CFITSIO packages: libcfitsio3 and libcfitsio3-dev
 
 6.  Define the PRESTO environment variable to the top level
   directory of the PRESTO distribution (i.e. this directory).
@@ -75,15 +73,17 @@ Basic steps to install:
   you will need to do a 'make mpi' as well.
 
 11.  The required libraries and miscellaneous files will be
-  located in $PRESTO/lib.  The executable will be in $PRESTO/bin.
+  located in $PRESTO/lib.  The executables will be in $PRESTO/bin.
   You may copy or move the executables wherever you like, but 
   the library files should stay put.  (That's why you define the 
   PRESTO variable -- so the routines can find them).
 
   Since dynamic linking will use $PRESTO/lib, you will need to make
   sure that that is in your LD_LIBRARY_PATH environment variable (or
   in some other way let your system know where to find the dynamic
-  libraries).  Similarly, you will want $PRESTO/bin in your PATH.
+  libraries).  Similarly, you will want $PRESTO/bin in your PATH, or
+  you can copy/link all of the files that are in $PRESTO/bin to
+  somewhere in your path.
 
 12.  If you want to save some disk space, do a 'make clean' in
   the 'src' directory.  This will leave the libraries and
@@ -92,21 +92,27 @@ Basic steps to install:
 
 13.  If you want to use all the python routines (which if you are
   doing anything but the most rudimentary analyses you will want 
-  to), you need Python >= version 2.7, and Numpy/Scipy
+  to), you need Python >= version 2.7 or >= 3.6, and Numpy/Scipy
   http://www.numpy.org  and  http://www.scipy.org
   In general, the following should work:
-     "cd $PRESTO/python ; make ; make fftfit"
-  If it doesnt, you will probably need to edit the Makefile 
-  or setup.py (one problem that is common is that ppgplot_libraries
-  sometimes needs "gfortran" added...).  Note that in order to
-  get python to find the modules and libraries, you should make
-  sure that your PYTHONPATH environment variable contains the
-  directory $PRESTO/lib/python.
+     "cd $PRESTO ; pip install ."
+  if you are using a virtual environment or have permissions for
+  installing normal Python packages.  If not, you may need to
+  install as an administrator or with the "--user" flag.  With
+  this new and better Python installation you do *not* need
+  to set the PYTHONPATH environment variable anymore and so you
+  should ensure that it doesn't contain $PRESTO/lib/python as in
+  the past.
+
+  If you want to test some of the Python code (especially the
+  trickiest bits to get compiled and installed), you can run:
+     "python tests/test_presto_python.py"
+  and you should get a bunch of tests and a bunch of "success"es.
 
 14.  Go find pulsars!
 
 Scott Ransom
-July 2019
+November 2019
 
 -----------------------------------------------------------------
 
@@ -120,17 +126,19 @@ compiling and running:
   -- Is TEMPO set to the top-level TEMPO source directory?
   -- Is PGPLOT_DIR set to the location of the PGPLOT utility files?
      (Note: On Ubuntu, that should be /usr/lib/pgplot5)
-  -- Is $PRESTO/bin in your PATH?
-  -- Is $PRESTO/lib in your LD_LIBRARY_PATH?
-  -- Is $PRESTO/lib/python in your PYTHONPATH?
+  -- Is $PRESTO/bin in your PATH? (It should be!)
+  -- Is $PRESTO/lib in your LD_LIBRARY_PATH? (It should be!)
+  -- Is $PRESTO/lib/python in your PYTHONPATH? (It shouldn't be now!)
 
 2.  Have you have installed the relevant -dev packages for glib2, FFTW
   and CFITSIO if you are using a Debian-based Linux distribution?
   Here are the required packages:
+    gfortran
     pgplot5
-	libcfitsio3-dev
+	libcfitsio-dev
 	libfftw3-dev
-	libpng12-dev
+	libpng-dev
+    libx11-dev
 	libglib2.0-dev
 
 3.  If you are using a Mac, Paul Ray has been running PRESTO a lot and
@@ -150,15 +158,10 @@ compiling and running:
    src/backend_common.c
 
    For python, you will need to install numpy, scipy, etc. using
-   MacPorts or pip, as you prefer.  Currently this must be Python 2.7,
-   but soon PRESTO will switch to Python 3.6 or later.
+   MacPorts or pip, as you prefer.
 
-   Next, edit python/Makefile to make sure that the version of f2py
-   you have installed is called.  Instead of f2py, you may need to
-   call f2py-2.7, for example.
-
-   Finally, in python/setup.py add "/opt/local/include" to
-   include_dirs.
+   Finally, in $PRESTO/setup.py you might need to add
+   "/opt/local/include" to the include_dirs.
 
    Then, just follow the PRESTO installation instructions (cd src;
-   make; make makewisdom; cd ../python ; make)
+   make; make makewisdom; cd .. ; pip install .)

README.md

@@ -3,19 +3,65 @@
 http://www.cv.nrao.edu/~sransom/presto/
 
 PRESTO is a large suite of pulsar search and analysis software
-developed by Scott Ransom mostly from scratch, and released under the
-GPL (v2).  It was primarily designed to efficiently search for binary
-millisecond pulsars from long observations of globular clusters
-(although it has since been used in several surveys with short
-integrations and to process a lot of X-ray data as well).  It is
-written primarily in ANSI C, with many of the recent routines in
+developed primarily by Scott Ransom mostly from scratch, and released
+under the GPL (v2).  It was primarily designed to efficiently search
+for binary millisecond pulsars from long observations of globular
+clusters (although it has since been used in several surveys with
+short integrations and to process a lot of X-ray data as well).  It
+is written primarily in ANSI C, with many of the recent routines in
 Python.  According to Steve Eikenberry, PRESTO stands for: PulsaR
 Exploration and Search TOolkit!
 
-**PRESTO has discovered over 600 pulsars, including more than 230
+**PRESTO has discovered over 700 pulsars, including almost 300
 recycled and/or binary pulsars!**
 
-## New in Version 2.1:
+## New in Version 3.0:
+ * This major release of PRESTO includes a massive restructuring
+   of python code and capabilities.  Things should work with Python
+   versions 2.7 and Python 3.6 and 3.7 at least.  The installation
+   of the python code has changed and has become more "pythonic"
+   so that PYTHONPATH is not needed, and all of the various modules
+   are now under a top-level "presto" module.  For example, to
+   use the psr_utils module you would now do:
+
+    import presto.psr_utils as pu
+
+   rather than
+
+    import psr_utils as pu
+
+   All of these changes will likely lead to code breakage and bugs!
+
+   Please check your code and processing carefully and post issues
+   (and hopefully pull requests) if you find them.
+
+   The installation instructions have been updated in the INSTALL file.
+
+   Huge thanks thanks go to **Gijs Molenaar, Matteo Bachetti, and
+   Paul Ray** for the work that they have done helping with this!
+
+ * There is also a new "examplescripts" directory where you will
+   find some example code to do a lot of important things, like
+   * Fully dedispersing an observation: `dedisp.py`
+   * Fully searching a dedispersed observation: `full_analysis.py`
+   * Sifting the results of a full search: `ACCEL_sift.py`
+   * Searching short chunks of a long time series: `short_analysis_simple.py`
+   * Making a really nice P-Pdot plane: `ppdot_plane_plot.py`
+   * and a few others.
+
+## Status of Version 2.2:
+ * Version 2.2 was the last version of PRESTO to work with the
+   old-style python interface which requires Python v2.7 or earlier
+   and is "installed" in-place and used via having $PRESTO/lib/python
+   in your PYTHONPATH.  There will probably be occasional bug fixes
+   for v2.2 in the `v2.2maint` branch of PRESTO.  You can get it
+   using:
+
+    git checkout -b v2.2maint origin/v2.2maint
+
+   and then installing as per the INSTALL file.
+
+## Improvements in Version 2.1:
  * `accelsearch` now has a "jerk" search capability (thanks to UVA
    undergrad Bridget Andersen for help with this!).  This makes
    searches take a *lot* longer, but definitely improves sensitivity
@@ -24,11 +70,6 @@ recycled and/or binary pulsars!**
    never need to set -zmax to anything larger than 300).
  * Ability to ignore bad channels on the command line (-ignorechan)
    (see `rfifind_stats.py` and `weights_to_ignorechan.py`)
- * Lots of new python utilities (such as for handling RFI, showing
-   bandpasses, making waterfall plots, ...)
- * New wrappers for the python interface (will make the transition
-   to Python 3.X much smoother later this year)
- * Many bug fixes and minor improvements
 
 ## About PRESTO:
 PRESTO is written with portability, ease-of-use, and memory efficiency
@@ -88,7 +129,7 @@ The Fourier-Domain acceleration search technique that PRESTO uses in
 the routine accelsearch is described in [Ransom, Eikenberry, and
 Middleditch
 (2002)](https://ui.adsabs.harvard.edu/abs/2002AJ....124.1788R/abstract),
-the new "jerk" search capability is described in
+the "jerk" search capability is described in
 [Andersen & Ransom (2018)](https://ui.adsabs.harvard.edu/abs/2018ApJ...863L..13A/abstract),
 and the phase-modulation search technique used by search_bin is described in
 [Ransom, Cordes, and Eikenberry
@@ -117,7 +158,7 @@ To update it on a regular basis do
     cd $PRESTO
     git pull
 
-and then re-make things in $PRESTO/src. 
+and then re-make things in $PRESTO/src.
 
 For more detailed installation instructions, see INSTALL.
 
@@ -129,7 +170,6 @@ If you want the "classic" branch, do the following:
 
     git clone git://github.com/scottransom/presto.git
     cd presto
-    git remote add classic origin/classic 
     git checkout -b classic origin/classic
 
 then build as per the (old) INSTALL file.
@@ -163,8 +203,8 @@ to process BCPM, SCAMP, and Spigot data, Jason Hessels for many
 contributions to the Python routines, and (alphabetical): Bridget
 Andersen, Anne Archibald, Cees Bassa, Matteo Bachetti, Slavko
 Bogdanov, Fernando Camilo, Paul Demorest, Paulo Freire, Chen Karako,
-Mike Keith, Patrick Lazarus, Maggie Livingstone, Chitrang Patel, Paul
-Ray, Paul Scholz, Ingrid Stairs, Kevin Stovall, Joeri van Leeuwen for
-many comments, suggestions and patches!
+Mike Keith, Patrick Lazarus, Maggie Livingstone, Gijs Molenaar,
+Chitrang Patel, Paul Ray, Paul Scholz, Ingrid Stairs, Kevin Stovall,
+Joeri van Leeuwen for many comments, suggestions and patches!
 
 Scott Ransom <[email protected]>

setup.py

@@ -7,7 +7,7 @@
 import setuptools
 from numpy.distutils.core import Extension, setup
 
-version = "2.2"
+version = "3.0"
 
 define_macros = []
 undef_macros = []