Tips and Traps¶
The easist way to define a UDF in PySpark is to use the
@udftag, and similarly the easist way to define a Pandas UDF in PySpark is to use the@pandas_udftag. Pandas UDFs are preferred to UDFs for server reasons. First, pandas UDFs are typically much faster than UDFs. Second, pandas UDFs are more flexible than UDFs on parameter passing. Both UDFs and pandas UDFs can take multiple columns as parameters. In addition, pandas UDFs can take a DataFrame as parameter (when passed to theapply
Column Functions and Operators in Spark
Convert PDF to EPS
There are tons of tools for converting PDF pictures to EPS pictures in Linux.
The pdf2ps command is a good one.
It produces EPS pictures without losing much resolution.
The general purpose tools convert
(from the ImageMagick package) does not produce as good quality EPS figures.
Terminal Multiplexers
-
There are 2 mature popular terminal multiplexer apps: screen and tmux. Both of them are very useful if you want to work on multiple tasks over 1 SSH connection. Screen is relative simple to use while tmux is much more powerful and more complicated to use.
-
Besides enabling users to …
Schedule Cron Tasks in a Docker Container
Cron tasks work in a Docker container.
However,
you have to manually start the cron deamon (root or sudo required) using cron or sudo cron
if it is not configured (via the Docker entrypoint) to start on the start of the Docker container.
For tutorials on crontab,
please refer to …
Regular Expression Equivalent
-
The order of precedence of operators in POSIX extended regular expression is as follows.
- Collation-related bracket symbols
[==],[::],[..] - Escaped characters
\ - Character set (bracket expression)
[] - Grouping
() - Single-character-ERE duplication
*,+,?,{m,n} - Concatenation
- Anchoring
^,$ - Alternation
|
- Collation-related bracket symbols
-
Some regular expression patterns are defined using a single leading backslash, e.g.,
\s,\b, etc. However, since special …