Categories
python

Import gotchas

  • Import problems can be tricky, since they reference many parts of the codebase and may cross into other modules or libraries. If you’re having difficulty with import, then something is wrong with your organization or how you’re importing the code. You may need to separate out different pieces of code (but sometimes they’re interdependent) and in general you need to organize code to reuse key pieces in other code with minimum reduplication of your effort. And you may need to have a single area to maintain and update key pieces of code – but there’s initialization and setup code also needed
  • import gotchas – from tips from Westra Chapter 7
    • name masking
      • If you name your package with the same name as a another package, then depending on the search order the interpreter will find one or the other package but you may introduce confusing difficult bugs.
      • If you name a python script with the same name as a different module, you can also mask the name and run into confusing difficulties. If you try to import the module you can end up erroneously importing your file.
      • The fix: you need to identify and give unique names, and you can use the package hierarchy to help
    • double import problems
      • sys.path issues
        • if we add a package directory to sys.path instead of the directory containing the package then the interpreter can import modules both directly and as part of the package, but the interpreter will initialize the module twice which can create subtle errors.
        • The fix: sys.path should include the directory containing the package and not any of its subdirectories
      • if you have executable top level run-once code in a script A and then write some additional functions in a script B that imports A, then the interpreter will run A twice.
        • The fix: it is best to move code that you want the interpreter run in an if __name__ == “__main__” block, which allows both
          • 1) the functions in the module to be imported without triggering the code and
          • 2) the module to be run at the command line
  • You can check out Westra’s Modular Programming book Chapter 7 for more on this, and sign up for more ideas on fixing complex bugs
Categories
python

Modular Programming with Python – review of first 3 chapters

Review Post – Modular Python by Erik Westra, Chapters 1-3

  • Functions are one of the most powerful tools in programming. As with any powerful tool, they also raise many questions and issues. Python has a specific terminology around functions, along with how to organize them into files that define Python modules. This book teaches firstly, Python modules as files that have Python functions, and secondly, modular programming in general. Modular programming is organizing large programs into pieces that you can reuse as requirements change. In this post I review how the book explains Python modules from the perspective of files of Python functions for those of us who want to learn how python modules work.
  • Chapter 1: Introducing Modular Programming
    • The introduction has some good basic goals. However the very first modular program uses very slightly advanced points that are irrelevant for the purpose of explaining modules, for example module-level global variables and exceptions. That said, for programmers who are not beginners, this is fine.
    • The next example, though somewhat simpler, similarly glosses over concepts which Westra hasn’t really introduced yet. The example packages a group of animal modules (toy Python functions in toy Python files) into an animal package, along with its initialization file – although we are still trying to learn what a module is . Next is an example of a larger program that might tell us why and how modularization improves our codebase and workflow.
    • Westra next shows us how to implement a module involving a cache. He gives us the definition of the module-level global variable we first saw, along with some public and private module functions. Overall this first chapter doesn’t teach us anything specific, and I recommend skimming it just to get an idea of his approach.
  • Chapter 2: Writing your first Modular Program
    • This chapter involves an inventory control system with modules for a report generator, data storage, and a user interface. The code covers a number of details that help us understand the size and scope of a program that begins to benefit from a modular approach.
    • Though seeing a big program is useful, the program again goes into many technical details that aren’t relevant to the modular programming concept – Westra acknowledges this and clearly says “don’t worry too much about the details of these functions”
    • In the implementation of the main program we see how to import the modules we created – this is the important point. This too is a chapter we can skim.
  • Chapter 3: Using Modules and Packages
    • Here we begin to actually learn how to create and use a directory (called a package) of Python module files.
    • From the outset Westra discusses packages within packages – this is a potentially confusing concept that he illustrates well, early, and consistently so that we can understand the principles clearly.
    • He diagrams directory trees and how these relate to packages which makes these clear. He next discusses initialization of modules – giving specific code examples, presented well and clearly.
    • After that, Westra goes into what the import statement does – likewise, specific, succinct and thorough.
    • Relative imports are likewise complex and so a major source of confusion. But again Westra diagrams the situation helpfully and clearly. He gives straightforward diagrams, and then writes the exact code that executes the depicted relative import. This method lets us quickly understand how the relative import system works.
Categories
python

While loops: 5 things to know

Number 1: what is a while loop:

Loops cause computers to repeat certain calculations in certain ways. While loops are one of the basic building blocks of programs. They can be a simple way to cause the program to repeat a particular set of instructions while a specific condition holds.

For example, we may want to repeatedly add numbers, sales or units, to find a total. While loops tell a computer to repeatedly perform such an instruction while a certain condition holds. Continuing the example, we may have a sales log file. Whenever a sale occurs, a program may add row a with the date, order number, quantity, price, and part number.

We may want to write a new program that instructs the
computer to read rows of the logfile and sum up the quantity*price. The while loop would help us total amount over the whole logfile, reading and summing while there are rows of data.

But we can do more complex tasks, such as finding sales up to Jan 1 of this year, by adding a condition to the while that the date is before jan 1 of this year. Similarly we can sum while the subtotal is below $20,000, which would tell us how long and what transactions let to that amount.

Though powerful, we face a learning curve when starting to write while loops. Here we cover 2 main categories of issues.

While Condition

Number 2: Never stopping

The while loop repeats the instructions while a certain condition holds, and stops repeating when that condition no longer holds. So the condition in the while loop needs to correctly capture all situations when the instructions are supposed to

  • run, and
  • stop

The classic error is the runaway while loop
here the stop is not reached:

i = 0
while i < 10:
    print(i)

Number 3: Never starting

Another classic is one where the loop never runs to begin with

i = 10
while i < 10:
    print(i)

Issues with updating

Number 4: updating issues

The other key issue is that the while loop typically needs to update some aspect of the data, such as a variable or a file
Often some arithmetic or complex logic is involved. These lead to many issues and need to be thoroughly understood while example with

i = 0
while i <= 10:
    print(i)
    i = i * 1.1

will never end because multiplication by 0 always results in 0.
Similarly

i = 0.1
while i <= sqrt(i):
    i = sqrt(i)
    print(i)

will never terminate because although i gets larger each time through the loop, it will never reach 1 since sqrt(x) < 1 for all x

Number 5: The off-by-1 error

The last is the off-by-1. Often loops have a small issue with the
update or with the while condition that causes errors that are
either 1 more or 1 less than the desired outcome in some way.
For example we may want to print 10 numbers but get 11:

i = 0
while i <= 10:
    print(i)
    i += 1

or we may get 9 numbers:

i = 1
while i < 10:
    print(i)
    i += 1

There are likewise 2 ways to get 10 numbers, only one of which might
be desired outcome

i = 1
while i <= 10:
    print(i)
    i += 1

or

i = 0
while i < 10:
    print(i)
    i += 1
Categories
python

How the python logger works in a few simple examples

this had also been a mystery to me. the python docs on the logger were pretty unintelligible, and i’ve used print statements for years, and i could see that the python logger seemed to be doing something, its just i couldn’t figure out what. so now the mystery is partially resolved. i think some examples might be the best way to learn about it. they added examples to the latest python docs which helped me alot, but i think some more might be useful.

the python logger is pretty intense, it’s designed to scale up to a project with multiple packages and modules. there’s a lot going on. to begin with, lets focus on a case with a single module


import logging
if __name__ = '__main__':  
    logging.basicConfig()
    mylogger = logging.getLogger('myfirstlogger')
    mylogger.debug('hello world')

this should setup a simple logger which does the equivalent of print. since mylogger is defined in the global scope, you can use it where you’d use a print statement. but it doesn’t give you any advantage over print. the first new thing you can do with logging is set the level. there’s predefined levels logging.DEBUG==10, logging.INFO=20, logging.WARNING=30, logging.ERROR=40, and logging.CRITICALERROR=50. you can generate a log entry at any of these levels with mylogger.info(‘whatever’) or mylogger.criticalerror(‘whatever’). however i use print statements mostly for debugging, so i am focused on logging.DEBUG and logging.INFO only. in fact, there’s inner loops which i sometimes want to debug but not always, and so i need levels lower than 10 for example. that’s easy, you can use mylogger(1, ‘whatever’) to have a level 1 message. and what would you do with a level 1 message? you can set the minimum level of the logger using setLevel:


import logging
if __name__ == '__main__':  
    logging.basicConfig()
    mylogger = logging.getLogger('myfirstlogger')
    mylogger.debug('hello world')
    mylogger.setLevel(5)
    mylogger.log(6, 'this will be printed')
    for i in range(10): mylogger.log(4, 'these will not since their level is too low')
    mylogger.setLevel(3)
    for i in range(10): mylogger.log(4,'but these will since we lowered the level')

pretty straightforward. however there’s one more feature of the logging system which makes it truly useful, which is that loggers form a hierarchy based on their names. notice the line mylogger=logging.getLogger(‘myfirstlogger’). this defines a top-level logger. we can create child loggers using mychildlogger = logging.getLogger(‘myfirstlogger.child’) for example. the level of the child logger, if unset, is inherited from the parent. but it can be set independently also.

each function in your module can have its own child logger, so you can turn on and off logging within each function, or have them relay the log messages up to the parent whenever you like.


import logging
def dothis():
    dothislogger = logging.getLogger('myfirstlogger.dothis')
    dothislogger.debug('wont be printed since we\'re inheriting the level from myfirstlogger')
    dothislogger.setLevel(logging.DEBUG)
    dothislogger.debug('will be printed')

if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    myfirstlogger = logging.getLogger('myfirstlogger')
    myfirstlogger.debug('wont be printed since the level is INFO')
    dothis()

so you can set whatever levels you need in all your child loggers, and they will relay relevant messages up to the parent loggers.

what i’ve covered so far gives us a lot of functionality over print statements — but there’s quite a lot more to explore in the logging system. it’s a full-industrial-strength system with quite a bit more features, some of which i don’t think i’ll ever need, but others which may come in handy.