Pyjo.Util - Portable utility functions

from Pyjo.Util import b64_encode, url_escape, url_unescape

string = 'test=23'
escaped = url_escape(string)
print(url_unescape(escaped))
print(b64_encode(escaped.encode(), ''))

Pyjo.Util provides portable utility functions for Pyjo.

Next to the functions of Mojolicious, it has functions of its own that turn statements into expressions, so that they can be used in a lambda, a callback or a chain: die(), getenv(), isalive(), notnone(), setenv(), slurp(), spurt() and warn().

home = getenv('HOME') or die('HOME is not defined')

See also

Pyjoyment, https://mojolicious.org.

Ported from Mojo::Util and Mojo::BaseUtil of Mojolicious 9.52.

Functions

class Pyjo.Util.Markup(*args, **kwargs)[source]
Pyjo.Util.b64_decode(b64: str | bytes) → bytes[source]
data = b64_decode(b64)

Base64 decode bytes, ignoring the characters that are not Base64 and the ones after the padding, as MIME::Base64 of Perl does.

Pyjo.Util.b64_encode(data: bytes, eol: str = '\n') → str[source]
b64 = b64_encode(data)
b64 = b64_encode(data, "\n")

Base64 encode bytes, the line ending defaults to a newline.

Pyjo.Util.b64url_decode(b64: str | bytes) → bytes[source]
data = b64url_decode(b64)

Base64 decode the variant for URLs of RFC 4648, with - and _ in the place of + and /, with or without the padding, as b64_decode() does, ignoring the characters that are not Base64.

# b"\xfb\xff\xfe"
b64url_decode('-__-')
Pyjo.Util.b64url_encode(data: bytes) → str[source]
b64 = b64url_encode(data)

Base64 encode bytes in the variant for URLs of RFC 4648, with - and _ in the place of + and /, and without the padding and the line endings.

# "-__-"
b64url_encode(b'\xfb\xff\xfe')
Pyjo.Util.camelize(string: str) → str[source]
camelcase = camelize(snakecase)

Convert snake_case string to CamelCase and replace - with ::.

# "FooBar"
camelize('foo_bar')

# "FooBar::Baz"
camelize('foo_bar-baz')

# "FooBar::Baz"
camelize('FooBar::Baz')
Pyjo.Util.class_to_file(class_: str) → str[source]
file = class_to_file('Foo.Bar')

Convert a class name to a file.

# "foo_bar"
class_to_file('Foo.Bar')

# "foobar"
class_to_file('FOO.Bar')

# "foo_bar"
class_to_file('FooBar')

# "foobar"
class_to_file('FOOBar')
Pyjo.Util.class_to_path(class_: str) → str[source]
path = class_to_path('Foo.Bar')

Convert class name to path, as a module is found in a directory of sys.path.

# "Foo/Bar.py"
class_to_path('Foo.Bar')

# "FooBar.py"
class_to_path('FooBar')
Pyjo.Util.decamelize(string: str) → str[source]
snakecase = decamelize(camelcase)

Convert CamelCase string to snake_case and replace :: with -.

# "foo_bar"
decamelize('FooBar')

# "foo_bar-baz"
decamelize('FooBar::Baz')

# "foo_bar-baz"
decamelize('foo_bar-baz')
Pyjo.Util.decode(charset: str, data: str | bytes) → str | None[source]
chars = decode('UTF-8', data)

Decode bytes to characters, or return None if decoding failed. A string of the characters 0-255 is decoded as the bytes it holds, like a Perl byte string; one with other characters does not decode.

Pyjo.Util.deprecated(*args: object) → None[source]
deprecated('foo is DEPRECATED in favor of bar')

Warn about deprecated feature from perspective of caller, with a DeprecationWarning. You can also set the PYJO_FATAL_DEPRECATIONS environment variable to make them raise it instead.

Pyjo.Util.die(e: BaseException | str) → NoReturn[source]
die(Exception('Something went wrong'))
die('Exit immediately')

Raise an exception. SystemExit exception is raised if parameter is not an exception already.

os.environ.get('HOME') or die('HOME is not defined')
Pyjo.Util.dumper(*values: object) → str[source]
python = dumper({'some': 'data'})

Dump a Python data structure with pprint.

Pyjo.Util.encode(charset: str, chars: str) → bytes[source]
data = encode('UTF-8', chars)

Encode characters to bytes. A character that the encoding cannot represent becomes a replacement character.

Pyjo.Util.extract_usage(file: str | PathLike[str] | None = None, section: str | None = None) → str[source]
usage = extract_usage()
usage = extract_usage('/home/sri/foo.py')
usage = extract_usage(section='myscript')

Extract usage message from the synopsis of the docstring of a module, its first literal block, defaults to using the file this function was called from.

# "Usage: APPLICATION test [OPTIONS]\n"
extract_usage()

"""
Test command
============
::

    Usage: APPLICATION test [OPTIONS]
"""

With a section, the first literal block after its heading, as the usage of a script that is a function of the module.

# "Usage: myscript [OPTIONS]\n"
extract_usage(section='myscript')

"""
Test module
===========
::

    test = Test.new()

myscript
--------
::

    Usage: myscript [OPTIONS]
"""
Pyjo.Util.generate_secret() → str[source]
secret = generate_secret()

Generate a random secret with a cryptographically secure random number generator.

Pyjo.Util.getenv(name: str) → str | None[source]
Pyjo.Util.getenv(name: str, default: str) → str
envvar = getenv(name)
envvar = getenv(name, default)

Get the environment variable. Returns default value or None if variable is not defined.

Pyjo.Util.getopt(specs: dict[str, Pyjo.GetOpt.Target], /, *, pass_through: bool = False, require_order: bool = False, env_prefix: str | None = None) → Pyjo.GetOpt.Pyjo_GetOpt_Result[source]
Pyjo.Util.getopt(args: list[str], specs: dict[str, Pyjo.GetOpt.Target], /, *, pass_through: bool = False, require_order: bool = False, env_prefix: str | None = None) → Pyjo.GetOpt.Pyjo_GetOpt_Result
opts = getopt({'H|headers=s': headers, 't|timeout=i': None, 'v|verbose': None})
opts = getopt(args, {'H|headers=s': headers, 't|timeout=i': None, 'v|verbose': None})
opts = getopt(args, {'H|headers=s': headers, 't|timeout=i': None}, pass_through=True)
opts = getopt(args, {'v|verbose': None}, require_order=True)
opts = getopt(args, {'t|timeout=i': None}, env_prefix='PYJO_')

Extract options from a list with Pyjo.GetOpt, defaults to using the arguments of sys.argv. The target of an option is a list for its values, a callback, which gets its name and its value, or None for its value in the result, which is false after an error. With require_order, the options end at the first argument that is not one. With env_prefix, an option that is not given has the value of the environment variable of its name, as PYJO_TIMEOUT for --timeout, see Pyjo.GetOpt.

# Extract "charset" option
opts = getopt(['--charset', 'UTF-8'], {'charset=s': None})
print(opts.charset)
Pyjo.Util.gunzip(compressed: bytes) → bytes[source]
uncompressed = gunzip(compressed)

Uncompress bytes with gzip.

Pyjo.Util.gzip(uncompressed: bytes) → bytes[source]
compressed = gzip(uncompressed)

Compress bytes with gzip.

Pyjo.Util.header_params(value: str) → tuple[dict[str, str], str][source]
params, remainder = header_params('one=foo; two="bar", three=baz')

Extract HTTP header field parameters until the first comma according to RFC 5987.

Pyjo.Util.hmac_sha1_bytes(data: bytes, secret: bytes) → bytes[source]
checksum = hmac_sha1_bytes(data, b'passw0rd')

Generate binary HMAC-SHA1 checksum for bytes with hmac.

Pyjo.Util.hmac_sha1_sum(data: bytes, secret: bytes) → str[source]
checksum = hmac_sha1_sum(data, b'passw0rd')

Generate HMAC-SHA1 checksum for bytes with hmac.

# "11cedfd5ec11adc0ec234466d8a0f2a83736aa68"
hmac_sha1_sum(b'foo', b'passw0rd')
Pyjo.Util.hmac_sha256_bytes(data: bytes, secret: bytes) → bytes[source]
checksum = hmac_sha256_bytes(data, b'passw0rd')

Generate binary HMAC-SHA256 checksum for bytes with hmac, as for a key derived in steps, each the key of the next one.

# The signing key of Signature Version 4 of AWS
key = b'AWS4' + secret
for step in (b'20150830', b'us-east-1', b'iam', b'aws4_request'):
    key = hmac_sha256_bytes(step, key)
Pyjo.Util.hmac_sha256_sum(data: bytes, secret: bytes) → str[source]
checksum = hmac_sha256_sum(data, b'passw0rd')

Generate HMAC-SHA256 checksum for bytes with hmac.

# "e54ce0e8e81828088bc8e81b9f2d96816e4a56ffe691d40f1426365e83623c8c"
hmac_sha256_sum(b'foo', b'passw0rd')
Pyjo.Util.hmac_sha512_bytes(data: bytes, secret: bytes) → bytes[source]
checksum = hmac_sha512_bytes(data, b'passw0rd')

Generate binary HMAC-SHA512 checksum for bytes with hmac.

Pyjo.Util.hmac_sha512_sum(data: bytes, secret: bytes) → str[source]
checksum = hmac_sha512_sum(data, b'passw0rd')

Generate HMAC-SHA512 checksum for bytes with hmac.

# "cd43bd8e691c5aff08152da9918221d91cc1c0f4cb63a80b5e1160387b98791f..."
hmac_sha512_sum(b'foo', b'passw0rd')
Pyjo.Util.html_attr_unescape(string: str) → str[source]
string = html_attr_unescape(escaped)

Same as html_unescape(), but handles special rules from the HTML Living Standard for HTML attributes.

# "foo=bar&ltest=baz"
html_attr_unescape('foo=bar&ltest=baz')

# "foo=bar<est=baz"
html_attr_unescape('foo=bar&lt;est=baz')
Pyjo.Util.html_unescape(string: str) → str[source]
string = html_unescape(escaped)

Unescape all HTML entities in string.

# "<div>"
html_unescape('&lt;div&gt;')
Pyjo.Util.humanize_bytes(size: int | float) → str[source]
string = humanize_bytes(1234)

Turn number of bytes into a simplified human readable format.

# "1B"
humanize_bytes(1)

# "7.5GiB"
humanize_bytes(8007188480)

# "13GiB"
humanize_bytes(13443399680)

# "-685MiB"
humanize_bytes(-717946880)
Pyjo.Util.isalive(obj: object) → bool[source]
boolean = isalive(weakref.proxy(obj))

Check if object behind weak reference proxy still exists.

Pyjo.Util.md5_bytes(data: bytes) → bytes[source]
checksum = md5_bytes(data)

Generate binary MD5 checksum for bytes with hashlib.

Pyjo.Util.md5_sum(data: bytes) → str[source]
checksum = md5_sum(data)

Generate MD5 checksum for bytes with hashlib.

# "acbd18db4cc2f85cedef654fccc4a4d8"
md5_sum(b'foo')
Pyjo.Util.monkey_patch(target: type | ModuleType, **patch: Callable[[...], Any]) → None[source]
monkey_patch(cls, foo=lambda: ...)
monkey_patch(module, foo=lambda: ...)
monkey_patch(cls, foo=lambda: ..., bar=lambda: ...)

Monkey patch functions into a class or a module, with names of their own.

monkey_patch(MyApp,
             one=lambda: print('One!'),
             two=lambda: print('Two!'),
             three=lambda: print('Three!'))
Pyjo.Util.network_contains(cidr: str, addr: str) → bool[source]
boolean = network_contains(network, address)

Check that a given address is contained within a network in CIDR form. If the network is a single address, the addresses must be equivalent.

# True
network_contains('10.0.0.0/8', '10.10.10.10')
network_contains('10.10.10.10', '10.10.10.10')
network_contains('fc00::/7', 'fc00::c0:ff:ee')

# False
network_contains('10.0.0.0/29', '10.10.10.10')
network_contains('10.10.10.12', '10.10.10.10')
network_contains('fc00::/7', '::1')
Pyjo.Util.notnone(*args: T | Callable[[], T] | None) → T | None[source]
value = notnone(value, another_value, default_value)
value = notnone(value, lambda: default_value)

Return the first argument that is not None. An argument that is callable is called, and its value is returned (lazy evaluation).

from Pyjo.Util import getenv

def print_log(msg, **kwargs):
    level = notnone(kwargs.get('log_level'), lambda: int(getenv('LOG_LEVEL', '3')))
    ...
Pyjo.Util.punycode_decode(punycode: str) → str[source]
string = punycode_decode(punycode)

Punycode decode string as described in RFC 3492.

# "bücher"
punycode_decode('bcher-kva')
Pyjo.Util.punycode_encode(string: str) → str[source]
punycode = punycode_encode(string)

Punycode encode string as described in RFC 3492.

# "bcher-kva"
punycode_encode('bücher')
Pyjo.Util.quote(string: str) → str[source]
Pyjo.Util.quote(string: bytes) → bytes
quoted = quote(string)

Quote string.

Pyjo.Util.random_bytes(count: int) → bytes[source]
random = random_bytes(count)

Generate count bytes worth of high entropy randomness.

Pyjo.Util.scope_guard(cb: Callable[[], object]) → _Guard[source]
with scope_guard(lambda: ...):
    ...

Create anonymous scope guard object that will execute the passed callback when the with block ends.

# Execute callback at end of block
with scope_guard(lambda: print('Mojo!')):
    print('Hello')
Pyjo.Util.secure_compare(one: str | bytes, two: str | bytes) → bool[source]
boolean = secure_compare(string1, string2)

Constant time comparison algorithm to prevent timing attacks. The secret string should be the second argument, to avoid leaking information about the length of the string.

Pyjo.Util.setenv(name: str, value: str | None) → None[source]
setenv(name, value)

Set the environment variable or remove it if value is None.

Pyjo.Util.sha1_bytes(data: bytes) → bytes[source]
checksum = sha1_bytes(data)

Generate binary SHA1 checksum for bytes with hashlib.

Pyjo.Util.sha1_sum(data: bytes) → str[source]
checksum = sha1_sum(data)

Generate SHA1 checksum for bytes with hashlib.

# "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
sha1_sum(b'foo')
Pyjo.Util.sha256_bytes(data: bytes) → bytes[source]
checksum = sha256_bytes(data)

Generate binary SHA256 checksum for bytes with hashlib.

Pyjo.Util.sha256_sum(data: bytes) → str[source]
checksum = sha256_sum(data)

Generate SHA256 checksum for bytes with hashlib.

# "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"
sha256_sum(b'foo')
Pyjo.Util.sha512_bytes(data: bytes) → bytes[source]
checksum = sha512_bytes(data)

Generate binary SHA512 checksum for bytes with hashlib.

Pyjo.Util.sha512_sum(data: bytes) → str[source]
checksum = sha512_sum(data)

Generate SHA512 checksum for bytes with hashlib.

# "f7fbba6e0636f890e56fbbf3283e524c6fa3204ae298382d624741d0dc663832..."
sha512_sum(b'foo')
Pyjo.Util.slugify(value: str | int, allow_unicode: bool = False) → str[source]
slug = slugify(string)
slug = slugify(string, boolean)

Returns a URL slug generated from the input string. Non-word characters are removed, the string is trimmed and lowercased, and whitespace characters are replaced by a dash. By default, non-ASCII characters are normalized to ASCII word characters or removed, but if a true value is passed as the second parameter, all word characters will be allowed in the result according to unicode semantics.

# "joel-is-a-slug"
slugify('Joel is a slug')

# "this-is-my-resume"
slugify('This is: my - résumé! ☃ ')

# "this-is-my-résumé"
slugify('This is: my - résumé! ☃ ', True)
Pyjo.Util.slurp(path: str | PathLike[str], charset: str = 'utf-8') → str[source]
string = slurp('/etc/passwd')
string = slurp('/etc/passwd', 'utf-8')

Read all data at once from file as characters. A byte order mark of Unicode switches utf-8, the default, to its encoding, and is not a part of the characters.

tree = split_cookie_header('a=b; expires=Thu, 07 Aug 2008 07:07:59 GMT')

Same as split_header(), but handles expires values from RFC 6265.

Pyjo.Util.split_header(string: str) → list[list[tuple[str, str | None]]][source]
tree = split_header('foo="bar baz"; test=123, yada')

Split HTTP header value into key/value pairs, each comma separated part gets its own list of (key, value) tuples, and keys without a value get None assigned.

# 'one'
split_header('one; two="three four", five=six')[0][0][0]

# 'two'
split_header('one; two="three four", five=six')[0][1][0]

# 'three four'
split_header('one; two="three four", five=six')[0][1][1]

# 'five'
split_header('one; two="three four", five=six')[1][0][0]

# 'six'
split_header('one; two="three four", five=six')[1][0][1]
Pyjo.Util.spurt(content: str, path: str | PathLike[str], charset: str = 'utf-8') → int[source]
written = spurt(string, '/etc/passwd')
written = spurt(string, '/etc/passwd', 'utf-8')

Write all characters at once to file, and return the number of bytes written.

Pyjo.Util.steady_time() → float[source]
time = steady_time()

High resolution time elapsed from an arbitrary fixed point in the past, resilient to time jumps if a monotonic clock is available through time.monotonic().

Pyjo.Util.tablify(rows: Sequence[Sequence[object]]) → str[source]
table = tablify([['foo', 'bar'], ['baz', 'yada']])

Row-oriented generator for text tables.

# "foo   bar\nyada  yada\nbaz   yada\n"
tablify([['foo', 'bar'], ['yada', 'yada'], ['baz', 'yada']])
Pyjo.Util.term_escape(string: str) → str[source]
escaped = term_escape(string)

Escape all POSIX control characters except for \n.

# "foo\\x09bar\\x0d\n"
term_escape("foo\tbar\r\n")
Pyjo.Util.trim(string: str) → str[source]
trimmed = trim(string)

Trim whitespace characters from both ends of string.

# "foo bar"
trim('  foo bar  ')
Pyjo.Util.unindent(string: str) → str[source]
unindented = unindent(string)

Unindent multi-line string.

# "foo\nbar\nbaz\n"
unindent("  foo\n  bar\n  baz\n")
Pyjo.Util.unquote(string: str) → str[source]
Pyjo.Util.unquote(string: bytes) → bytes
string = unquote(quoted)

Unquote string.

Pyjo.Util.url_escape(string: str, pattern: str | None = None) → str[source]
Pyjo.Util.url_escape(string: bytes, pattern: bytes | None = None) → bytes
escaped = url_escape(string)
escaped = url_escape(string, r'^A-Za-z0-9\-._~')

Percent encode unsafe characters in string as described in RFC 3986, the pattern used defaults to ^A-Za-z0-9\-._~.

# "foo%3Bbar"
url_escape('foo;bar')
Pyjo.Util.url_unescape(string: str) → str[source]
Pyjo.Util.url_unescape(string: bytes) → bytes
string = url_unescape(escaped)

Decode percent encoded characters in string as described in RFC 3986. A percent encoded byte in a string becomes the character with the same code, as in Perl.

# "foo;bar"
url_unescape('foo%3Bbar')
Pyjo.Util.warn(*args: object) → None[source]
warn(string)
warn(string, value)

Output the arguments to sys.stderr, joined as Perl’s warn joins a list, with a newline at the end unless there is one already.

Pyjo.Util.xml_escape(string: str | Markup | None) → str[source]
escaped = xml_escape(string)

Escape unsafe characters &, <, >, " and ' in string, but do not escape markup: an object with the method __html__, as Pyjo.String.HTML, which gives the markup.

# "&lt;div&gt;"
xml_escape('<div>')

# "<div>"
from Pyjo.String.HTML import html
xml_escape(html('<div>'))
Pyjo.Util.xor_encode(data: bytes, key: bytes) → bytes[source]
encoded = xor_encode(data, key)

XOR encode string with variable length key.