Mercurial > prosody-modules

annotate mod_firewall/definitions.lib.lua @ 2522:72cbec103709

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mod_firewall: Improve HTTP polling logic
author Matthew Wild <mwild1@gmail.com>
date Mon, 20 Feb 2017 09:26:16 +0000
parents c6fd8975704b
children a3a18d09ae8a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2079
edec9de0220a mod_firewall: Silence warnings about unused arguments [luacheck]
Kim Alvefur <zash@zash.se>
parents: 1863
diff changeset
1
edec9de0220a mod_firewall: Silence warnings about unused arguments [luacheck]
Kim Alvefur <zash@zash.se>
parents: 1863
diff changeset
2 -- Name arguments are unused here
edec9de0220a mod_firewall: Silence warnings about unused arguments [luacheck]
Kim Alvefur <zash@zash.se>
parents: 1863
diff changeset
3 -- luacheck: ignore 212
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
4
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
5 local definition_handlers = {};
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
6
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
7 local http = require "net.http";
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
8 local timer = require "util.timer";
1863
92602cfac751 mod_firewall: Fix missing import of util.set (used to be global)
Kim Alvefur <zash@zash.se>
parents: 999
diff changeset
9 local set = require"util.set";
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
10 local new_throttle = require "util.throttle".create;
2128
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
11
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
12 local multirate_cache_size = module:get_option_number("firewall_multirate_cache_limit", 1000);
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
13
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
14 function definition_handlers.ZONE(zone_name, zone_members)
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
15 local zone_member_list = {};
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
16 for member in zone_members:gmatch("[^, ]+") do
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
17 zone_member_list[#zone_member_list+1] = member;
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
18 end
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
19 return set.new(zone_member_list)._items;
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
20 end
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
21
2128
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
22 -- Helper function used by RATE handler
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
23 local function evict_only_unthrottled(name, throttle)
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
24 throttle:update();
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
25 -- Check whether the throttle is at max balance (i.e. totally safe to forget about it)
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
26 if throttle.balance < throttle.max then
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
27 -- Not safe to forget
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
28 return false;
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
29 end
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
30 end
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
31
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
32 function definition_handlers.RATE(name, line)
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
33 local rate = assert(tonumber(line:match("([%d.]+)")), "Unable to parse rate");
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
34 local burst = tonumber(line:match("%(%s*burst%s+([%d.]+)%s*%)")) or 1;
2128
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
35 local max_throttles = tonumber(line:match("%(%s*entries%s+([%d]+)%s*%)")) or multirate_cache_size;
2370
5fe483b73fd2 mod_firewall: Rate limiting: Document 'entries' and add option to allow overflowing when full
Matthew Wild <mwild1@gmail.com>
parents: 2131
diff changeset
36 local deny_when_full = not line:match("%(allow overflow%)");
2128
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
37 return {
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
38 single = function ()
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
39 return new_throttle(rate*burst, burst);
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
40 end;
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
41
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
42 multi = function ()
2370
5fe483b73fd2 mod_firewall: Rate limiting: Document 'entries' and add option to allow overflowing when full
Matthew Wild <mwild1@gmail.com>
parents: 2131
diff changeset
43 local cache = require "util.cache".new(max_throttles, deny_when_full and evict_only_unthrottled or nil);
2128
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
44 return {
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
45 poll_on = function (_, key, amount)
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
46 assert(key, "no key");
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
47 local throttle = cache:get(key);
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
48 if not throttle then
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
49 throttle = new_throttle(rate*burst, burst);
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
50 if not cache:set(key, throttle) then
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
51 module:log("warn", "Multirate '%s' has hit its maximum number of active throttles (%d), denying new events", name, max_throttles);
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
52 return false;
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
53 end
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
54 end
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
55 return throttle:poll(amount);
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
56 end;
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
57 }
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
58 end;
21bc4d7cddae mod_firewall: Add support for throttling based on user-defined properties (experimental)
Matthew Wild <mwild1@gmail.com>
parents: 2079
diff changeset
59 };
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
60 end
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
61
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
62 local list_backends = {
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
63 memory = {
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
64 init = function (self, type, opts)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
65 if opts.limit then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
66 local have_cache_lib, cache_lib = pcall(require, "util.cache");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
67 if not have_cache_lib then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
68 error("In-memory lists with a size limit require Prosody 0.10");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
69 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
70 self.cache = cache_lib.new((assert(tonumber(opts.limit), "Invalid list limit")));
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
71 if not self.cache.table then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
72 error("In-memory lists with a size limit require a newer version of Prosody 0.10");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
73 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
74 self.items = self.cache:table();
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
75 else
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
76 self.items = {};
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
77 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
78 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
79 add = function (self, item)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
80 self.items[item] = true;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
81 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
82 remove = function (self, item)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
83 self.items[item] = nil;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
84 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
85 contains = function (self, item)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
86 return self.items[item] == true;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
87 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
88 };
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
89 http = {
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
90 init = function (self, url, opts)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
91 local poll_interval = assert(tonumber(opts.ttl or "3600"), "invalid ttl for <"..url.."> (expected number of seconds)");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
92 local pattern = opts.pattern or "([^\r\n]+)\r?\n";
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
93 assert(pcall(string.match, "", pattern), "invalid pattern for <"..url..">");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
94 if opts.hash then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
95 assert(opts.hash:match("^%w+$") and type(hashes[opts.hash]) == "function", "invalid hash function: "..opts.hash);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
96 self.hash_function = hashes[opts.hash];
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
97 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
98 local etag;
2522
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
99 local failure_count = 0;
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
100 local retry_intervals = { 60, 120, 300 };
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
101 local function update_list()
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
102 http.request(url, {
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
103 headers = {
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
104 ["If-None-Match"] = etag;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
105 };
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
106 }, function (body, code, response)
2522
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
107 local next_poll = poll_interval;
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
108 if code == 200 and body then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
109 etag = response.headers.etag;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
110 local items = {};
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
111 for entry in body:gmatch(pattern) do
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
112 items[entry] = true;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
113 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
114 self.items = items;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
115 module:log("debug", "Fetched updated list from <%s>", url);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
116 elseif code == 304 then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
117 module:log("debug", "List at <%s> is unchanged", url);
2522
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
118 elseif code == 0 or (code >= 400 and code <=599) then
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
119 module:log("warn", "Failed to fetch list from <%s>: %d %s", url, code, tostring(body));
2522
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
120 next_poll = 300;
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
121 failure_count = failure_count + 1;
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
122 next_poll = retry_intervals[failure_count] or retry_intervals[#retry_intervals];
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
123 end
2522
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
124 if next_poll > 0 then
72cbec103709 mod_firewall: Improve HTTP polling logic
Matthew Wild <mwild1@gmail.com>
parents: 2520
diff changeset
125 timer.add_task(next_poll+math.random(0, 60), update_list);
2520
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
126 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
127 end);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
128 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
129 update_list();
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
130 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
131 contains = function (self, item)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
132 if self.hash_function then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
133 item = self.hash_function(item);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
134 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
135 return self.items and self.items[item] == true;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
136 end;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
137 };
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
138 };
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
139
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
140 local function create_list(list_backend, list_def, opts)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
141 if not list_backends[list_backend] then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
142 error("Unknown list type '"..list_backend.."'", 0);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
143 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
144 local list = setmetatable({}, { __index = list_backends[list_backend] });
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
145 if list.init then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
146 list:init(list_def, opts);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
147 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
148 return list;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
149 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
150
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
151 --[[
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
152 %LIST spammers: memory (source: /etc/spammers.txt)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
153
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
154 %LIST spammers: memory (source: /etc/spammers.txt)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
155
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
156
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
157 %LIST spammers: http://example.com/blacklist.txt
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
158 ]]
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
159
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
160 function definition_handlers.LIST(list_name, list_definition)
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
161 local list_backend = list_definition:match("^%w+");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
162 local opts = {};
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
163 local opt_string = list_definition:match("^%S+%s+%((.+)%)");
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
164 if opt_string then
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
165 for opt_k, opt_v in opt_string:gmatch("(%w+): ?([^,]+)") do
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
166 opts[opt_k] = opt_v;
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
167 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
168 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
169 return create_list(list_backend, list_definition:match("^%S+"), opts);
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
170 end
c6fd8975704b mod_firewall: Initial support for lists, in-memory and HTTP
Matthew Wild <mwild1@gmail.com>
parents: 2370
diff changeset
171
999
197af8440ffb mod_firewall: Make defining objects generic (currently zones and rate limits), so more can easily be added. Also a syntax change... definition lines must begin with %
Matthew Wild <mwild1@gmail.com>
parents:
diff changeset
172 return definition_handlers;