Implement --blacklist option and include more modules/recipes by default

Jonas Thiem · Jonas Thiem · commit 3b40ba01bb43 · 2019-02-06T17:20:58.000+01:00
* Adds `--blacklist` option that prevents recipes/packages from being
  added even when they are in the `depends` of recipes or otherwise added

* Makes the following modules included per default: `android` (via
  `bootstrap.py` dependency of all bootstraps),
  `openssl`/`libffi`/`sqlite3` (via `python3` recipe dependency)

* Documents `--blacklist` option and that `android` is now included
  by default

* Cleans up the packaging kivy/sdl2 apps part in the Quickstart section
diff --git a/doc/source/apis.rst b/doc/source/apis.rst
@@ -12,8 +12,8 @@ Runtime permissions
 With API level >= 21, you will need to request runtime permissions
 to access the SD card, the camera, and other things.
 
-This can be done through the `android` module, just add it to
-your `--requirements` (as `android`) and then use it in your app like this::
+This can be done through the `android` module which is *available per default*
+unless you blacklist it. Use it in your app like this::
 
       from android.permissions import request_permission, Permission
       request_permission(Permission.WRITE_EXTERNAL_STORAGE)
@@ -34,8 +34,8 @@ longer than necessary (with your app already being loaded) due to a
 limitation with the way we check if the app has properly started.
 In this case, the splash screen overlaps the app gui for a short time.
 
-To dismiss the loading screen explicitely in your code, add p4a's `android`
-module to your `--requirements` and use this::
+To dismiss the loading screen explicitely in your code, use the `android`
+module::
 
   from android import hide_loading_screen
   hide_loading_screen()
@@ -92,14 +92,14 @@ Under SDL2, you can handle the `appropriate events <https://wiki.libsdl.org/SDL_
 Advanced Android API use
 ------------------------
 
+.. _reference-label-for-android-module:
+
 `android` for Android API access
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mentioned above, the ``android`` Python module provides a simple 
-wrapper around many native Android APIS, and it can be included by
-adding it to your requirements, e.g. :code:`--requirements=kivy,android`.
-It is not automatically included by Kivy unless you use the old (Pygame)
-bootstrap.
+wrapper around many native Android APIS, and it is *included per default*
+unless you blacklist it.
 
 The available functionality of this module is not separately documented.
 You can read the source `on
@@ -136,7 +136,7 @@ This is obviously *much* less verbose than with Pyjnius!
 
 
 `Pyjnius` - raw lowlevel API access
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Pyjnius lets you call the Android API directly from Python Pyjnius is
 works by dynamically wrapping Java classes, so you don't have to wait
diff --git a/doc/source/buildoptions.rst b/doc/source/buildoptions.rst
@@ -243,3 +243,25 @@ options (this list may not be exhaustive):
 - ``add-source``: Add a source directory to the app's Java code.
 - ``--compile-pyo``: Optimise .py files to .pyo.
 - ``--resource``: A key=value pair to add in the string.xml resource file.
+
+
+Blacklist (APK size optimization)
+---------------------------------
+
+To optimize the size of the `.apk` file that p4a builds for you,
+you can **blacklist** certain core components. Per default, p4a
+will add python *with batteries included* as would be expected on
+desktop, including openssl, sqlite3 and other components you may
+not use.
+
+To blacklist an item, specify the ``--blacklist`` option::
+
+    p4a apk ... --blacklist=sqlite3
+
+At the moment, the following core components can be blacklisted
+(if you don't want to use them) to decrease APK size:
+
+- ``android``  disables p4a's android module (see :ref:`reference-label-for-android-module`)
+- ``libffi``  disables ctypes stdlib module
+- ``openssl``   disables ssl stdlib module
+- ``sqlite3``   disables sqlite3 stdlib module
diff --git a/doc/source/quickstart.rst b/doc/source/quickstart.rst
@@ -149,18 +149,34 @@ You have the possibility to configure on any command the PATH to the SDK, NDK an
 Usage
 -----
 
-Build a Kivy application
-~~~~~~~~~~~~~~~~~~~~~~~~
+Build a Kivy or SDL2 application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To build your application, you need to have a name, version, a package
-identifier, and explicitly write the bootstrap you want to use, as
-well as the requirements::
+To build your application, you need to specify name, version, a package
+identifier, the bootstrap you want to use (`sdl2` for kivy or sdl2 apps)
+and the requirements::
+
+    p4a apk --private $HOME/code/myapp --package=org.example.myapp --name "My application" --version 0.1 --bootstrap=sdl2 --requirements=python3,kivy
+
+**Note on `--requirements`: you must add all
+libraries/dependencies your app needs to run.**
+Example: `--requirements=python3,kivy,vispy`. For an SDL2 app,
+`kivy` is not needed, but you need to add any wrappers you might
+use (e.g. `pysdl2`).
+
+This `p4a apk ...` command builds a distribution with `python3`,
+`kivy`, and everything else you specified in the requirements.
+It will be packaged using a SDL2 bootstrap, and produce
+an `.apk` file.
 
-    p4a apk --private $HOME/code/myapp --package=org.example.myapp --name "My application" --version 0.1 --bootstrap=sdl2 --requirements=python2,kivy
+*Compatibility notes:*
 
-This will first build a distribution that contains `python2` and `kivy`, and using a SDL2 bootstrap. Python2 is here explicitely written as kivy can work with python2 or python3.
+- While python2 is still supported by python-for-android,
+  it will possibly no longer receive patches by the python creators
+  themselves in 2020. Migration to Python 3 is recommended!
 
-You can also use ``--bootstrap=pygame``, but this bootstrap is deprecated for use with Kivy and SDL2 is preferred.
+- You can also use ``--bootstrap=pygame``, but this bootstrap
+  is deprecated and not well-tested.
 
 Build a WebView application
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -171,46 +187,34 @@ well as the requirements::
 
     p4a apk --private $HOME/code/myapp --package=org.example.myapp --name "My WebView Application" --version 0.1 --bootstrap=webview --requirements=flask --port=5000
 
+**Please note as with kivy/SDL2, you need to specify all your
+additional requirements/depenencies.**
+
 You can also replace flask with another web framework.
 
 Replace ``--port=5000`` with the port on which your app will serve a
 website. The default for Flask is 5000.
 
-Build an SDL2 based application
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This includes e.g. `PySDL2
-<https://pysdl2.readthedocs.io/en/latest/>`__.
-
-To build your application, you need to have a name, version, a package
-identifier, and explicitly write the sdl2 bootstrap, as well as the
-requirements::
-
-    p4a apk --private $HOME/code/myapp --package=org.example.myapp --name "My SDL2 application" --version 0.1 --bootstrap=sdl2 --requirements=your_requirements
-
-Add your required modules in place of ``your_requirements``,
-e.g. ``--requirements=pysdl2`` or ``--requirements=vispy``.
-
 Other options
 ~~~~~~~~~~~~~
 
 You can pass other command line arguments to control app behaviours
 such as orientation, wakelock and app permissions. See
 :ref:`bootstrap_build_options`.
 
-    
+
 
 Rebuild everything
 ~~~~~~~~~~~~~~~~~~
 
 If anything goes wrong and you want to clean the downloads and builds to retry everything, run::
 
     p4a clean_all
-    
+
 If you just want to clean the builds to avoid redownloading dependencies, run::
 
     p4a clean_builds && p4a clean_dists
-    
+
 Getting help
 ~~~~~~~~~~~~
 
@@ -269,7 +273,7 @@ You can list the available distributions::
 And clean all of them::
 
     p4a clean_dists
-    
+
 Configuration file
 ~~~~~~~~~~~~~~~~~~
 
diff --git a/pythonforandroid/bootstrap.py b/pythonforandroid/bootstrap.py
@@ -52,6 +52,7 @@ class Bootstrap(object):
     # All bootstraps should include Python in some way:
     recipe_depends = [
         ("python2", "python2legacy", "python3", "python3crystax"),
+        'android',
     ]
 
     can_be_chosen_automatically = True
diff --git a/pythonforandroid/graph.py b/pythonforandroid/graph.py
@@ -39,10 +39,13 @@ def conflicts(self):
         return False
 
 
-def get_dependency_tuple_list_for_recipe(recipe, blacklist=[]):
+def get_dependency_tuple_list_for_recipe(recipe, blacklist=None):
     """ Get the dependencies of a recipe with filtered out blacklist, and
         turned into tuples with fix_deplist()
     """
+    if blacklist is None:
+        blacklist = set()
+    assert(type(blacklist) == set)
     if recipe.depends is None:
         dependencies = []
     else:
@@ -51,21 +54,26 @@ def get_dependency_tuple_list_for_recipe(recipe, blacklist=[]):
 
         # Filter out blacklisted items and turn lowercase:
         dependencies = [
-            deptuple for deptuple in dependencies
-            if not set(deptuple).intersection(set(blacklist))
+            tuple(set(deptuple) - blacklist)
+            for deptuple in dependencies
+            if tuple(set(deptuple) - blacklist)
         ]
     return dependencies
 
 
 def recursively_collect_orders(
-        name, ctx, all_inputs, orders=[], blacklist=[]
+        name, ctx, all_inputs, orders=None, blacklist=None
         ):
     '''For each possible recipe ordering, try to add the new recipe name
     to that order. Recursively do the same thing with all the
     dependencies of each recipe.
 
     '''
     name = name.lower()
+    if orders is None:
+        orders = []
+    if blacklist is None:
+        blacklist = set()
     try:
         recipe = Recipe.get_recipe(name, ctx)
         dependencies = get_dependency_tuple_list_for_recipe(
@@ -75,7 +83,8 @@ def recursively_collect_orders(
         # handle opt_depends: these impose requirements on the build
         # order only if already present in the list of recipes to build
         dependencies.extend(fix_deplist(
-            [[d] for d in recipe.get_opt_depends_in_list(all_inputs)]
+            [[d] for d in recipe.get_opt_depends_in_list(all_inputs)
+             if d.lower() not in blacklist]
         ))
 
         if recipe.conflicts is None:
@@ -106,7 +115,9 @@ def recursively_collect_orders(
             dependency_new_orders = [new_order]
             for dependency in dependency_set:
                 dependency_new_orders = recursively_collect_orders(
-                    dependency, ctx, all_inputs, dependency_new_orders)
+                    dependency, ctx, all_inputs, dependency_new_orders,
+                    blacklist=blacklist
+                )
 
             new_orders.extend(dependency_new_orders)
 
@@ -132,14 +143,16 @@ def find_order(graph):
                 bset.discard(result)
 
 
-def obvious_conflict_checker(ctx, name_tuples, blacklist=[]):
+def obvious_conflict_checker(ctx, name_tuples, blacklist=None):
     """ This is a pre-flight check function that will completely ignore
         recipe order or choosing an actual value in any of the multiple
         choice tuples/dependencies, and just do a very basic obvious
         conflict check.
     """
     deps_were_added_by = dict()
     deps = set()
+    if blacklist is None:
+        blacklist = set()
 
     # Add dependencies for all recipes:
     to_be_added = [(name_tuple, None) for name_tuple in name_tuples]
@@ -227,7 +240,7 @@ def obvious_conflict_checker(ctx, name_tuples, blacklist=[]):
     return None
 
 
-def get_recipe_order_and_bootstrap(ctx, names, bs=None, blacklist=[]):
+def get_recipe_order_and_bootstrap(ctx, names, bs=None, blacklist=None):
     # Get set of recipe/dependency names, clean up and add bootstrap deps:
     names = set(names)
     if bs is not None and bs.recipe_depends:
@@ -236,7 +249,9 @@ def get_recipe_order_and_bootstrap(ctx, names, bs=None, blacklist=[]):
         ([name] if not isinstance(name, (list, tuple)) else name)
         for name in names
     ])
-    blacklist = [bitem.lower() for bitem in blacklist]
+    if blacklist is None:
+        blacklist = set()
+    blacklist = {bitem.lower() for bitem in blacklist}
 
     # Remove all values that are in the blacklist:
     names_before_blacklist = list(names)
@@ -261,7 +276,9 @@ def get_recipe_order_and_bootstrap(ctx, names, bs=None, blacklist=[]):
         new_possible_orders = [RecipeOrder(ctx)]
         for name in name_set:
             new_possible_orders = recursively_collect_orders(
-                name, ctx, name_set, orders=new_possible_orders)
+                name, ctx, name_set, orders=new_possible_orders,
+                blacklist=blacklist
+            )
         possible_orders.extend(new_possible_orders)
 
     # turn each order graph into a linear list if possible
@@ -305,7 +322,8 @@ def get_recipe_order_and_bootstrap(ctx, names, bs=None, blacklist=[]):
                 "Could not find any compatible bootstrap!"
             )
         recipes, python_modules, bs = get_recipe_order_and_bootstrap(
-            ctx, chosen_order, bs=bs)
+            ctx, chosen_order, bs=bs, blacklist=blacklist
+        )
     else:
         # check if each requirement has a recipe
         recipes = []
diff --git a/pythonforandroid/recipes/python3/__init__.py b/pythonforandroid/recipes/python3/__init__.py
@@ -23,7 +23,7 @@ class Python3Recipe(GuestPythonRecipe):
 
     patches = ["patches/fix-ctypes-util-find-library.patch"]
 
-    depends = ['hostpython3']
+    depends = ['hostpython3', 'sqlite3', 'openssl', 'libffi']
     conflicts = ['python3crystax', 'python2', 'python2legacy']
 
     configure_args = (
diff --git a/pythonforandroid/toolchain.py b/pythonforandroid/toolchain.py
@@ -171,16 +171,24 @@ def build_dist_from_args(ctx, dist, args):
     """Parses out any bootstrap related arguments, and uses them to build
     a dist."""
     bs = Bootstrap.get_bootstrap(args.bootstrap, ctx)
-    build_order, python_modules, bs \
-        = get_recipe_order_and_bootstrap(ctx, dist.recipes, bs)
+    blacklist = getattr(args, "blacklist", "").split(",")
+    if len(blacklist) == 1 and blacklist[0] == "":
+        blacklist = []
+    build_order, python_modules, bs = (
+        get_recipe_order_and_bootstrap(
+            ctx, dist.recipes, bs,
+            blacklist=blacklist
+        ))
     ctx.recipe_build_order = build_order
     ctx.python_modules = python_modules
 
     info('The selected bootstrap is {}'.format(bs.name))
     info_main('# Creating dist with {} bootstrap'.format(bs.name))
     bs.distribution = dist
-    info_notify('Dist will have name {} and recipes ({})'.format(
+    info_notify('Dist will have name {} and requirements ({})'.format(
         dist.name, ', '.join(dist.recipes)))
+    info('Dist contains the following requirements as recipes: {}'.format(
+        ctx.recipe_build_order))
     info('Dist will also contain modules ({}) installed from pip'.format(
         ', '.join(ctx.python_modules)))
 
@@ -301,6 +309,13 @@ def __init__(self):
                   'Python modules'),
             default='')
 
+        generic_parser.add_argument(
+            '--blacklist',
+            help=('Blacklist an internal recipe from use. Allows '
+                  'disabling Python 3 core modules to save size'),
+            dest="blacklist",
+            default='')
+
         generic_parser.add_argument(
             '--bootstrap',
             help='The bootstrap to build with. Leave unset to choose '
@@ -448,7 +463,6 @@ def add_parser(subparsers, *args, **kwargs):
             help='Symlink the dist instead of copying')
 
         parser_apk = add_parser(
-
             subparsers,
             'apk', help='Build an APK',
             parents=[generic_parser])
@@ -527,9 +541,11 @@ def add_parser(subparsers, *args, **kwargs):
         if args.debug:
             logger.setLevel(logging.DEBUG)
 
-        # strip version from requirements, and put them in environ
+        # Process requirements and put version in environ
         if hasattr(args, 'requirements'):
             requirements = []
+
+            # Parse --requirements argument list:
             for requirement in split_argument_list(args.requirements):
                 if "==" in requirement:
                     requirement, version = requirement.split(u"==", 1)
diff --git a/tests/test_graph.py b/tests/test_graph.py

Original file line number	Diff line number	Diff line change
`@@ -52,6 +52,7 @@ class Bootstrap(object):`
`52`	`52`	`# All bootstraps should include Python in some way:`
`53`	`53`	`recipe_depends = [`
`54`	`54`	`("python2", "python2legacy", "python3", "python3crystax"),`
	`55`	`+ 'android',`
`55`	`56`	`]`
`56`	`57`
`57`	`58`	`can_be_chosen_automatically = True`