Welcome! Log In Create A New Profile

Advanced

"variables" in configuration

Posted by pacman 
"variables" in configuration
September 25, 2017 01:10PM
Intro / background (feel free to ignore) ...
First off, I'm very new to nginx. I've been using Apache for many years (since 1997 I think), but started moving to nginx a few days ago.

I had a go at it around a month ago, but due to too many difficulties, I gave up.
Since a friend convinced med that nginx is the way to go, I tried again and got it working with Perl CGI scripts, so I can move my servers.

The suggestion ...
As I'm new, I think it's important to catch my first-time experience.
Since I have many sites on Apache, I'd like to simplify configuration and make it short.
Fortunately, nginx is made in such a way that normal configuration is real short.
-But if you have 60 web-sites that is configured in almost the same way (apart from the subdomain or domain name and a few environment variables), then it's cumbersome to create a configuration file for each site.
Generating configurations for each site also introduces a lot of redundancy; now you have two configuration files per site; one to feed to the script for defining the actual configuration and one output from the script, which is fed into the server.
The configuration files that are fed into the server are of course 'expanded' and thus use a little more space than the templates. Also it's confusing having two configuration files per site. Debugging and fixing errors is a little more complicated than just having one file, which is expanded using the built-in 'include' directive.

As many others, I started looking at using 'set' for shortening the configurations.
I know that at this moment you're thinking: "This is wrong", and you're perfectly right.
Variables are evaluated at runtime, which means they'll slow down the performance of the server.

-But what if we had something similar to variables.
-Something close to '#undef + #define' in the C-pre-processor.

Such 'macros' need only be processed whenever nginx is reloading the configuration.
They would need to be applied for each 'include' directive (treating the configuration itself as an include file).
-Thus they would be applied before any actual parsing of the configuration occur (as I expect that include is; I haven't looked at the sources, though).

The directive could be named 'macro', 'string', 'const', 'constant', 'absolute', 'replace' or whatever else you'd like.
Personally I think that 'string' would be the best name, because it would be a string, but it would be allowed to re-define it after first time it's defined.
It could have almost the same syntax as the 'set' directive, but the way it would work is quite different.
The directive searches through the text and replaces any occurrences of the found 'name' by the parameter given as 'value'.
Let's say the chosen name for the directive is 'string' and that it has the following syntax:
string name value;

Each time such a 'string' directive is encountered, the macro-expander is applied on the value.
After that, the value is placed in a key-value dictionary (and of course ... name is the key, the expanded value is the value).
Whenever a '$' is encountered, the name would be looked up in the dictionary. If it's found, it'll be replaced with the corresponding value.

Pseudo-code (sorry, the indentation didn't make it):
for all lines {
current = here; /* save current position */
(token, name, value) = nextToken(here);
if(token == "}")
{
dictionary = pop();
}
else if(token == "{")
{
push(dictionary); /* duplicates it */
}
else if(token == "include")
{
insertFile(name, here); /* oversimplified for this example. */
}
else if(token[0] == '$')
{
value = dictionary.lookup(token);
if(value)
{
here = replace(current, name, value); /* replace the found token by the value and point 'here' after the inserted string */
}
}
else if(token == "string")
{
value = macroExpand(dictionary, value);
dictionary.add(name, value);
}
}

The example given below is using the '$' prefix; I am aware that this is probably not a good prefix; I've done this for making it easy to see where I'm going. The prefix could be anything that is 'available' as prefix.

in http section:
# The server root; I'm calling it $home; as it would make sense to set it to nginx's home directory.
string $home "/www";

in server section:
string $domain example.com;
include /etc/nginx/snippets/each-site.conf;

in snippets/each-site.conf:

string $site $home/$domain;
string $cgi_root $site/CGI-Executables;
string $cgi_data $site/CGI-Data;
string $cgi_data $site/Sites;

server {
server_name $domain www.$domain;
access_log $site/Logs/access.log;
error_log $site/Logs/error.log;
...
... other settings ...
...
}

I would of course very much like a 'string' directive like mentioned above. I do not know if it's possible or if there are things, that would make it difficult to implement in nginx.

Another solution could be to add parameters/arguments to the include directive:
include /etc/nginx/snippets/each-site.conf /www example.com Sites CGI-Executables CGI-Data;
... and define "macros" using $1, $2, $3, $4 ... like in Bash. It would get the job done, but I think it would be less elegant.
The advantage would be that it reduces the configuration per site to one line each.

I know there's been many people asking about how to use variables for the configuration (for instance in server_name), and I know that it's not over; there will be many more as time passes.
I believe the solution is to have something that's parsed once only during loading the configuration - like a simple 'find' / 'replace' feature.
The suggested macro-expander is simple, light-weight and yet quite powerful. I'm using a similar one in some of my own applications and have grown quite fond of using it.
Re: "variables" in configuration
September 25, 2017 02:05PM
... Actually ... It might be a good idea to allow quoting too:

string $name "value which includes a ';' so that multiple macros can be nested";
string $name 'value with apostrophs as alternative to quotes; apostrophs will not allow "inner" macro-expansion';
... in addition to ...
string $name value;

My above post suggests macro-expanding the value before adding it to the dictionary.
This is correct behaviour, but one thing I forgot to mention is that ...
1: All values that are not found in the dictionary must keep their '$name', because they would be normal variables.
2: if '$name' itself is found, then it would cause infinite recursion; this should of course be checked in order to avoid it.
3: If an apostroph is used around a value, do not parse it first, but if a value is 'bare' or enclosed in double-quotes, it should be parsed before adding it to the dictionary.

During processing of the file, I've written the following comment:
/* replace the found token by the value and point 'here' after the inserted string */
-This should not be so; instead set 'here' to the beginning of the inserted string, in order to allow parsing the result.


This last modification will allow us to make even more powerful configuration-macros ...

string $configure_site 'string $site $home/$domain; string $cgi_root $site/CGI-Executables; $cgi_data $site/CGI-Data; include /etc/nginx/snippets/site-common.conf';

string $domain example.com; $configure_site;
string $domain example.org; $configure_site;
string $domain example.net; $configure_site;
Re: "variables" in configuration
September 27, 2017 12:44AM
I spent some time making an example as proof-of-concept.
This is written in nodejs, so in order to try it out, you'll need that installed.
In short, if you don't have nodejs installed already ...

sudo apt-get install nodejs npm; npm install fs colors

Run the example by typing:

./t

Note: The pre-processor is junk. The MacroExpander class is a little more usable.
Attachments:
open | download - macro-expander.tar.bz2 (2.9 KB)
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 156
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready