<a id="main-features"></a>
## Main Features
-* Support of the following fully pre-configured domain blocklist feeds (free for private usage, for commercial use please check their individual licenses)
+Support of the following fully pre-configured domain blocklist feeds (free for private usage, for commercial use please check their individual licenses)
| Feed | Enabled | Size | Focus | Information |
| :------------------ | :-----: | :--- | :--------------- | :-------------------------------------------------------------------------------- |
| winspy | | S | win_telemetry | [Link](https://github.com/crazy-max/WindowsSpyBlocker) |
| yoyo | | S | general | [Link](https://pgl.yoyo.org/adservers) |
+**Please note:** Feeds marked with size **VAR** (`1Hosts`, `hagezi`, `ipfire_dbl`, `stevenblack`, `utcapitole`) additionally require a category selection via the options `adb_hst_feed`, `adb_hag_feed`, `adb_ipf_feed`, `adb_stb_feed` and `adb_utc_feed` (or via the LuCI feed configuration). Without a category the feed is skipped during processing.
+
* List of supported and fully pre-configured adblock sources, already active sources are pre-selected.
<b><em>To avoid OOM errors, please do not select too many lists!</em></b>
List size information with the respective domain ranges as follows:
* Provides a detailed DNS Report with DNS related information about client requests, top (blocked) domains and more
* Provides a powerful search function to quickly find blocked (sub-)domains, e.g. to allow certain domains
* Implements a jail mode - only domains on the allowlist are permitted, all other DNS requests are rejected
-* Automatic blocklist backup & restore, these backups will be used in case of download errors and during startup
+* Automatic blocklist backup & restore: backups are used on `start`/`restart` and as a fallback on download errors — feeds are only actually refreshed via `reload`
* Send notification E-Mails, see example configuration below
* Add new adblock feeds on your own with the `Custom Feed Editor` in LuCI or via CLI, see example below
* Strong LuCI support, all relevant options are exposed to the web frontend
* For performance reasons, adblock depends on gnu sort and gawk
* Before update from former adblock releases please make a backup of your local allow- and blocklists. In the latest adblock these lists have been renamed to `/etc/adblock/adblock.allowlist` and `/etc/adblock/adblock.blocklist`. There is no automatic content transition to the new files.
* The uci configuration of adblock is automatically migrated during package installation via the uci-defaults mechanism using a housekeeping script
+* Only `reload` actually refreshes the feeds (ETag check plus download of changed feeds). `start`, `restart` — and `boot`/`resume` — restore the existing blocklist backups and only download feeds that have **no** backup yet; they do **not** re-fetch already cached feeds. To update your blocklists (e.g. from a cron job) always use `reload`
+
<a id="installation-and-usage"></a>
## Installation & Usage
| adb_nftbridge | -, not set | enables a temporary DNS bridge to an external DNS resolver during local DNS restarts |
| adb_bridgednsv4 | -, not set | external IPv4 DNS resolver used during bridging |
| adb_bridgednsv6 | -, not set | external IPv6 DNS resolver used during bridging |
+| adb_hst_feed | -, not set | category selection for the `1hosts` feed (required to enable it) |
+| adb_hag_feed | -, not set | category selection for the `hagezi` feed (required to enable it) |
+| adb_ipf_feed | -, not set | category selection for the `ipfire_dbl` feed (required to enable it) |
+| adb_stb_feed | -, not set | category selection for the `stevenblack` feed (required to enable it) |
+| adb_utc_feed | -, not set | category selection for the `utcapitole` feed (required to enable it) |
<a id="examples"></a>
## Examples
This additional firewall feature lets selected client devices temporarily bypass local DNS blocking and use an external, unfiltered DNS resolver. It is designed for situations where a device needs short‑term access to content normally blocked by the adblock rules.
A lightweight CGI endpoint handles the workflow:
-* The client opens the URL, e.g. http(s)://\<ROUTER-IP\>cgi-bin/adblock (preferably transferred via QR code shown in LuCI)
+* The client opens the URL, e.g. http(s)://\<ROUTER-IP\>/cgi-bin/adblock (preferably transferred via QR code shown in LuCI)
* The script automatically detects the device’s MAC address
* If the MAC is authorized, the script displays the current status:
* Not in the nftables set → option to request a temporary allow (“Bypass”)
By default adblock uses the following pre-configured download options:
```
- * curl: --connect-timeout 20 --retry-delay 10 --retry 4 --retry-all-errors --fail --silent --show-error --location -o
- * wget: --no-cache --no-cookies --timeout=20 --waitretry=10 --tries=5 --retry-connrefused --max-redirect=0 -O
+ * curl: --connect-timeout 20 --retry-delay 10 --retry 4 --retry-max-time 80 --retry-all-errors --fail --silent --show-error --location -o
+ * wget: --no-cache --no-cookies --timeout=20 --waitretry=10 --tries=5 --retry-connrefused -O
* uclient-fetch: --timeout=20 -O
```
Finally enable E-Mail support, add a valid E-Mail receiver address in LuCI and setup an appropriate cron job.
**Automatic adblock feed updates and E-Mail reports**
-For a regular, automatic update of the used feeds or other regular adblock tasks set up a cron job. In LuCI you find the cron settings under `System` => `Scheduled Tasks`. On the command line the cron file is located at `/etc/crontabs/root`:
+For a regular, automatic update of the used feeds or other regular adblock tasks set up a cron job. Use `reload` here — `start`/`restart` would only restore the backups instead of fetching fresh feeds. In LuCI you find the cron settings under `System` => `Scheduled Tasks`. On the command line the cron file is located at `/etc/crontabs/root`:
Example 1
```sh
*[!a-zA-Z0-9_]*) ;;
*)
- eval "${option}=\"\${value}\""
+ [ -n "${value}" ] && eval "${option}=\"\${value}\""
;;
esac
}
f_fetch() {
local fetch fetch_list insecure update="0"
+ # check if the configured fetch utility is available and has SSL support,
+ # if not try to find an alternative with SSL support or log an error if not found
+ #
adb_fetchcmd="$(command -v "${adb_fetchcmd}" 2>/dev/null)"
if [ -z "${adb_fetchcmd}" ]; then
fetch_list="curl wget-ssl libustream-openssl libustream-wolfssl libustream-mbedtls"
esac
done
fi
-
[ -z "${adb_fetchcmd}" ] && f_log "err" "download utility with SSL support not found, please set 'adb_fetchcmd' manually"
+ # check the fetch retry value
+ #
+ case "${adb_fetchretry}" in
+ 0* | *[!0-9]*)
+ adb_fetchretry="5"
+ ;;
+ esac
+
+ # set fetch parameters based on the fetch utility and check if insecure fetching is enabled
+ #
case "${adb_fetchcmd##*/}" in
"curl")
[ "${adb_fetchinsecure}" = "1" ] && insecure="--insecure"
# handle etag http header
#
f_etag() {
- local http_head http_code etag_id etag_cnt etag_match out_rc="4" feed="${1}" feed_url="${2}" feed_suffix="${3}" feed_cnt="${4:-"1"}"
+ local http_head http_code etag_id etag_cnt etag_match out_rc="4" feed="${1}" feed_url="${2}" feed_suffix="${3}" feed_cnt="${4:-"1"}" feed_rm="${5:-"0"}"
if [ -n "${adb_etagparm}" ]; then
#
[ ! -f "${adb_backupdir}/adblock.etag" ] && : >"${adb_backupdir}/adblock.etag"
- # fetch http headers and extract http code and etag/last-modified header
- #
- http_head="$("${adb_fetchcmd}" ${adb_etagparm} "${feed_url}${feed_suffix}" 2>&1)"
- http_code="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*http\/[0-9.]+ /{printf "%s",$2}')"
- etag_id="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*etag: /{gsub("\"","");printf "%s",$2}')"
+ if [ "${feed_rm}" = "0" ]; then
- # if etag header is not present, try to use last-modified header as fallback for change detection
- #
- if [ -z "${etag_id}" ]; then
- etag_id="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*last-modified: /{gsub(/[Ll]ast-[Mm]odified:|[[:space:]]|,|:/,"");printf "%s\n",$1}')"
+ # fetch http headers and extract http code and etag/last-modified header
+ #
+ http_head="$("${adb_fetchcmd}" ${adb_etagparm} "${feed_url}${feed_suffix}" 2>&1)"
+ http_code="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*http\/[0-9.]+ /{printf "%s",$2}')"
+ etag_id="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*etag: /{gsub(/[\r"]/,"");printf "%s",$2}')"
+
+ # if etag header is not present, try to use last-modified header as fallback for change detection
+ #
+ if [ -z "${etag_id}" ]; then
+ etag_id="$(printf '%s' "${http_head}" | "${adb_awkcmd}" 'tolower($0)~/^[[:space:]]*last-modified: /{gsub(/[Ll]ast-[Mm]odified:|[[:space:]]|,|:/,"");printf "%s\n",$1}')"
+ fi
fi
# acquire exclusive lock on etag file to serialize concurrent read-modify-write from parallel feeds
exec 9>"${adb_etaglock}"
"${adb_flockcmd}" -x 9
- # compare http code and etag id with stored values, update etag file and return code accordingly
- #
- etag_cnt="$("${adb_awkcmd}" -v f="${feed}" -v s="${feed_suffix}" -v e="${etag_id}" '
- BEGIN { matched = 0; cnt = 0; p = f " " s }
- $1 == f { cnt++ }
- index($0, p) == 1 {
- rest = substr($0, length(p) + 1)
- sub(/^[[:space:]]+/, "", rest)
- if (rest == e) { matched = 1 }
- }
- END { print cnt; exit !matched }' "${adb_backupdir}/adblock.etag")"
- etag_match="${?}"
+ if [ "${feed_rm}" = "0" ]; then
+
+ # compare http code and etag id with stored values, update etag file and return code accordingly
+ #
+ etag_cnt="$("${adb_awkcmd}" -v f="${feed}" -v s="${feed_suffix}" -v e="${etag_id}" '
+ BEGIN { matched = 0; cnt = 0; p = f " " s }
+ $1 == f { cnt++ }
+ index($0, p) == 1 {
+ rest = substr($0, length(p) + 1)
+ sub(/^[[:space:]]+/, "", rest)
+ if (rest == e) { matched = 1 }
+ }
+ END { print cnt; exit !matched }' "${adb_backupdir}/adblock.etag")"
+ etag_match="${?}"
+ fi
# compare http code, etag count and etag id; update etag file and return code accordingly
#
- if [ "${http_code}" = "200" ] && [ "${etag_cnt}" = "${feed_cnt}" ] && [ -n "${etag_id}" ] && [ "${etag_match}" = "0" ]; then
+ if [ "${feed_rm}" = "0" ] && [ "${http_code}" = "200" ] && [ "${etag_cnt}" = "${feed_cnt}" ] && [ -n "${etag_id}" ] && [ "${etag_match}" = "0" ]; then
out_rc="0"
- elif [ -n "${etag_id}" ]; then
+ elif [ -n "${etag_id}" ] || [ "${feed_rm}" = "1" ]; then
# if feed count is less than etag count, it means the feed source has been removed or disabled, so remove all entries for this feed,
# otherwise only remove the entry with the matching feed suffix (feed url) to allow multiple sources for the same feed
#
- if [ "${feed_cnt}" -lt "${etag_cnt}" ]; then
+ if [ "${feed_rm}" = "1" ] || [ "${feed_cnt}" -lt "${etag_cnt:-"0"}" ]; then
"${adb_awkcmd}" -v f="${feed}" '$1 != f' \
"${adb_backupdir}/adblock.etag" >"${adb_backupdir}/adblock.etag.new"
else
"${adb_backupdir}/adblock.etag" >"${adb_backupdir}/adblock.etag.new"
fi
"${adb_mvcmd}" -f "${adb_backupdir}/adblock.etag.new" "${adb_backupdir}/adblock.etag"
- printf '%s\t%s\n' "${feed} ${feed_suffix}" "${etag_id}" >>"${adb_backupdir}/adblock.etag"
+ [ "${feed_rm}" = "0" ] && printf '%s\t%s\n' "${feed} ${feed_suffix}" "${etag_id}" >>"${adb_backupdir}/adblock.etag"
out_rc="2"
fi
exec 9>&-
fi
- f_log "debug" "f_etag ::: feed: ${feed}, suffix: ${feed_suffix:-"-"}, http_code: ${http_code:-"-"}, feed/etag: ${feed_cnt}/${etag_cnt:-"0"}, rc: ${out_rc}"
+ f_log "debug" "f_etag ::: feed: ${feed}, suffix: ${feed_suffix:-"-"}, http_code: ${http_code:-"-"}, feed/etag: ${feed_cnt}/${etag_cnt:-"0"}, rm: ${feed_rm}, rc: ${out_rc}"
return "${out_rc}"
}
fi
}
+# remove a single feed from the active feed list
+#
+f_feedrm() {
+ adb_feed=" ${adb_feed} "
+ adb_feed="${adb_feed// ${1} / }"
+ adb_feed="${adb_feed# }"
+ adb_feed="${adb_feed% }"
+}
+
# backup/restore/remove blocklists
#
f_list() {
f_list backup
elif [ "${adb_action}" != "boot" ] && [ "${adb_action}" != "start" ]; then
f_log "info" "preparation of '${src_name}' failed, rc: ${src_rc}"
+ [ "${adb_action}" = "reload" ] && f_etag "${src_name}" "" "" "" "1"
f_list restore
out_rc="${?}"
: >"${src_tmpfile}"
else
f_log "info" "download of '${src_name}' failed, url: ${src_url}, rule: ${src_rset:-"-"}, categories: ${src_cat:-"-"}, rc: ${src_rc}"
if [ "${adb_action}" != "boot" ] && [ "${adb_action}" != "start" ]; then
+ [ "${adb_action}" = "reload" ] && f_etag "${src_name}" "" "" "" "1"
f_list restore
out_rc="${?}"
fi
;;
"backup")
file_name="${src_tmpfile}"
+ "${adb_rmcmd}" -f "${src_tmprmfile}"
"${adb_gzipcmd}" -cf "${src_tmpfile}" >"${adb_backupdir}/adb_list.${src_name}.gz"
out_rc="${?}"
;;
fi
case "${adb_action}" in
"boot" | "start" | "restart" | "resume") ;;
-
*)
if [ -n "${src_name}" ] && [ "${out_rc}" != "0" ]; then
- adb_feed=" ${adb_feed} "
- adb_feed="${adb_feed// ${src_name} / }"
- adb_feed="${adb_feed# }"
- adb_feed="${adb_feed% }"
+ printf '%s\n' "${src_name}" >"${src_tmprmfile}"
fi
;;
esac
"remove")
"${adb_rmcmd}" "${adb_backupdir}/adb_list.${src_name}.gz" 2>>"${adb_errorlog}"
out_rc="${?}"
- adb_feed=" ${adb_feed} "
- adb_feed="${adb_feed// ${src_name} / }"
- adb_feed="${adb_feed# }"
- adb_feed="${adb_feed% }"
+ f_feedrm "${src_name}"
;;
"merge")
src_name=""
adb_cnt="0"
feeds="restrictive jail (allowlist-only)"
else
- feeds="$(printf '%s\n' ${adb_feed// /, } | "${adb_sortcmd}" | "${adb_xargscmd}")"
+ feeds="$(printf '%s\n' ${adb_feed} | "${adb_sortcmd}" | "${adb_xargscmd}")"
fi
fi
fi
if [ -n "${log_msg}" ] && { [ "${class}" != "debug" ] || [ "${adb_debug}" = "1" ]; }; then
if [ -x "${adb_loggercmd}" ]; then
- "${adb_loggercmd}" -p "${class}" -t "adblock-${adb_bver:-"-"}[${$}]" "${log_msg::512}"
+ "${adb_loggercmd}" -p "${class}" -t "adblock-${adb_bver:-"n/a"}[${$}]" "${log_msg::512}"
else
- printf '%s %s %s\n' "${class}" "adblock-${adb_bver:-"-"}[${$}]" "${log_msg::512}" >&2
+ printf '%s %s %s\n' "${class}" "adblock-${adb_bver:-"n/a"}[${$}]" "${log_msg::512}" >&2
fi
if [ "${class}" = "err" ] || [ "${class}" = "emerg" ]; then
[ "${adb_action}" != "mail" ] && f_rmdns
#
f_main() {
local src_name src_domain src_rset src_url src_cat src_item src_list src_entries src_suffix src_rc entry cnt
- local src_tmpcat src_tmparchive src_tmpload src_tmpfile seen_domains feed_restore map_domain
+ local src_tmpcat src_tmparchive src_tmpload src_tmprmfile src_tmpfile rm_file seen_domains feed_restore map_domain
# allow- and blocklist preparation
#
# check if feed is defined in configuration, if not remove it from feed list and continue with next one
#
if ! json_select "${src_name}" >/dev/null 2>&1; then
- adb_feed=" ${adb_feed} "
- adb_feed="${adb_feed// ${src_name} / }"
- adb_feed="${adb_feed# }"
- adb_feed="${adb_feed% }"
+ f_feedrm "${src_name}"
continue
fi
src_tmpcat="${adb_tmpload}.${src_name}.cat"
src_tmpload="${adb_tmpload}.${src_name}.load"
src_tmparchive="${adb_tmpload}.${src_name}.archive"
+ src_tmprmfile="${adb_tmpload}.${src_name}.remove"
src_tmpfile="${adb_tmpfile}.${src_name}"
src_rc=4
done
wait
+ # prune feeds that failed restore inside the background subshells
+ #
+ for rm_file in "${adb_tmpload}".*.remove; do
+ [ -f "${rm_file}" ] || continue
+ while read -r entry; do
+ f_feedrm "${entry}"
+ done <"${rm_file}"
+ done
+
# tld compression and dns restart
#
if f_list merge && [ -s "${adb_tmpdir}/${adb_dnsfile}" ]; then
top_clients)
"${adb_sortcmd}" ${adb_srtopts} -nr "${top_tmpclients}" |
"${adb_awkcmd}" -v top_count="${top_count}" '
+ function jclean(s){gsub(/[[:cntrl:]"\\]/,"",s);return s}
BEGIN { ORS=""; OFS="" }
NR==1 {
- printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
NR>1 && NR<=top_count {
- printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
' >>"${report_jsn}"
;;
top_domains)
"${adb_sortcmd}" ${adb_srtopts} -nr "${top_tmpdomains}" |
"${adb_awkcmd}" -v top_count="${top_count}" '
+ function jclean(s){gsub(/[[:cntrl:]"\\]/,"",s);return s}
BEGIN { ORS=""; OFS="" }
NR==1 {
- printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
NR>1 && NR<=top_count {
- printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
' >>"${report_jsn}"
;;
top_blocked)
"${adb_sortcmd}" ${adb_srtopts} -nr "${top_tmpblocked}" |
"${adb_awkcmd}" -v top_count="${top_count}" '
+ function jclean(s){gsub(/[[:cntrl:]"\\]/,"",s);return s}
BEGIN { ORS=""; OFS="" }
NR==1 {
- printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf "\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
NR>1 && NR<=top_count {
- printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, $2
+ printf ",\n\t\t{\n\t\t\t\"count\": \"%s\",\n\t\t\t\"address\": \"%s\"\n\t\t}", $1, jclean($2)
}
' >>"${report_jsn}"
;;
i = 0
printf "\t\"requests\": [\n"
}
+ function jclean(s){gsub(/[[:cntrl:]"\\]/,"",s);return s}
# only match if search is empty or non-empty and NF == 7
((search == "" || index($0, search)) && NF == 7) {
printf "\n\t\t{\n"
printf "\t\t\t\"date\": \"%s\",\n", $1
printf "\t\t\t\"time\": \"%s\",\n", $2
- printf "\t\t\t\"client\": \"%s\",\n", $3
- printf "\t\t\t\"iface\": \"%s\",\n", $4
+ printf "\t\t\t\"client\": \"%s\",\n", jclean($3)
+ printf "\t\t\t\"iface\": \"%s\",\n", jclean($4)
printf "\t\t\t\"type\": \"%s\",\n", $5
- printf "\t\t\t\"domain\": \"%s\",\n", $6
+ printf "\t\t\t\"domain\": \"%s\",\n", jclean($6)
printf "\t\t\t\"rc\": \"%s\"\n", $7
printf "\t\t}"
}
git.99rst.org / openwrt-packages.git / commitdiff