Powell Unified Framework – Code Conventions

Some Python code conventions that make it easier to map between the mathematics and the code

Powell Unified Framework
Reinforcement Learning
Python
Author

Kobus Esterhuysen

Published

April 21, 2023

Python identifier conventions for alignment with the Powell Unified Framework

Background

Dr. Warren Powell’s unified framework shows great promise for unifying the formalisms of at least a dozen different fields. Using his framework enables easier access to thinking patterns in these other fields that might be beneficial and informative to the sequential decision problem at hand. Traditionally, this kind of problem would be approached from the reinforcement learning perspective. However, using Dr. Powell’s wider and more comprehensive perspective almost certainly provides additional value.

Here is information on Dr. Powell’s perspective on Sequential Decision Analytics.

Dr. Powell places strong emphasis on having a consistent and well thought out notation - to him it is akin to a language that, once you understand it, unlocks all the potential benefits of sequential decision analytics. It is best to standardize this language among the communities. He works through a complete example in chapter 9 of his impressive textbook Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions. Section 9.2 Notational Style, in particular, lays down some valuable conventions.

Motivation

In order for me to have a strong mapping between Python code that follows Dr. Powell’s Unified Framework (PUF) and the mathematics from the unified framework, I find it useful to follow the conventions mentioned below. In the software community it is a best practice to use descriptive Python identifier (i.e. variable) names. However, I tend to lean towards the more ‘cryptic’ (i.e. mathematical) way of naming things in order to gain a less ‘noisy’ mapping between the mathematics and the code.

I experimented with making use of Greek symbols as well as super/sub scripts in notebooks but, until now, have not had consistent success with that, in particular when I tried to do this in Google Colab notebooks. Maybe things will be much easier in this regard with the Julia language.

Mapping

It is helpful to consider my mapping in the context of chapter 9, and in particular, Section 9.2 Notational Style of Dr. Powells book. Here is a summary of my mapping between the mathematics and the corresponding variable names in the Python code:

  • Superscripts
    • variable names have a double underscore to indicate a superscript
    • \(X^{\pi}\): has code X__pi, is read ‘X pi’
  • Subscripts
    • variable names have a single underscore to indicate a subscript
    • \(S_t\): has code S_t, is read ‘S at t’
    • \(M^{Spend}_t\) has code M__Spend_t which is read: ‘M Spend at t’
  • Arguments
    • collection variable names may have argument information added
    • \(X^{\pi}(S_t)\): has code X__piIS_tI, is read ‘X pi in S at t’
    • the surrounding I’s are used to imitate the parentheses around the argument
  • Next time/iteration
    • variable names that indicate one step in the future are quite common
    • \(R_{t+1}\): has code R_tt1, is read ‘R at t+1’
    • \(R^{n+1}\): has code R__nt1, is read ‘R at n+1’
  • Rewards
    • State-independent terminal reward and cumulative reward
      • \(F\): has code F for terminal reward
      • \(\sum_{n}F\): has code cumF for cumulative reward
    • State-dependent terminal reward and cumulative reward
      • \(C\): has code C for terminal reward
      • \(\sum_{t}C\): has code cumC for cumulative reward
  • Vectors where components use different names
    • \(S_t(R_t, p_t)\): has code S_t.R_t and S_t.p_t, is read ‘S at t in R at t, and, S at t in p at t’
    • the code implementation is by means of a named tuple
      • self.State = namedtuple('State', SNames) for the ‘class’ of the vector
      • self.S_t for the ‘instance’ of the vector
  • Vectors where components reuse names
    • \(x_t(x_{t,GB}, x_{t,BL})\): has code x_t.x_t_GB and x_t.x_t_BL, is read ‘x at t in x at t for GB, and, x at t in x at t for BL’
    • the code implementation is by means of a named tuple
      • self.Decision = namedtuple('Decision', xNames) for the ‘class’ of the vector
      • self.x_t for the ‘instance’ of the vector
  • Use of mixed-case variable names
    • to reduce confusion, sometimes the use of mixed-case variable names are preferred (even though it is not a best practice in the Python community), reserving the use of underscores and double underscores for math-related variables

Pronunciation

I find it very helpful to read the mathematical variables by pronouncing them. It is important (to me) to do this in a well-defined way and consistently. Here are a few conventions in this regard:

  • Flavors
    • \(M^{Spend}\) has code M__Spend which is read: ‘M Spend’
    • Note that the ‘flavor’ is orally combined with the main name, i.e. it is read ‘M Spend’ and not ‘M super Spend’
  • Time/Iteration indices
    • \(R_{t+1}\): has code R_tt1, is read ‘R at t+1’
    • \(R^{n+1}\): has code R__nt1, is read ‘R at n+1’
    • Note the preposition at which always precedes the mention of the index
  • Function arguments
    • \(S_t(R_t)\): is read ‘S at t in R at t’
    • Note the preposition in which always precedes the mention of the argument(s)
  • Vectors where components reuse names
    • \((x_{t,GB}, x_{t,BL})\): is read ‘x at t for GB, and, x at t for BL’
    • Note the preposition for for the different components of the vector

Conclusion

Admittedly, the above scheme, if you are not comfortable with the PUF, makes it harder to read and understand the code. However, once you have mastered the conventions in the PUF, it should make the code much easier to digest. In the future, we may be able to reliably make use of natural superscripts and subscripts in notebooks as well as Greek symbols which would make the mapping even less noisy.