Mercurial > ec-dotfiles

annotate bin/moefetch @ 520:a60af1e859e4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
set PATH before calling uname.
author edogawaconan <me@myconan.net>
date Fri, 06 Jun 2014 10:49:10 +0900
parents 38c7615caf9e
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
221
e891b563b797 wrong rule caused mass headache
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 220
diff changeset
1 #!/bin/sh
148
edhoprima
parents:
diff changeset
2
301
36bc27bb32ff Updated version and copyright.
Edho Arief <edho@myconan.net>
parents: 300
diff changeset
3 # Copyright (c) 2009-2012, edogawaconan <edho@myconan.net>
148
edhoprima
parents:
diff changeset
4 #
edhoprima
parents:
diff changeset
5 # Permission to use, copy, modify, and/or distribute this software for any
edhoprima
parents:
diff changeset
6 # purpose with or without fee is hereby granted, provided that the above
edhoprima
parents:
diff changeset
7 # copyright notice and this permission notice appear in all copies.
edhoprima
parents:
diff changeset
8 #
edhoprima
parents:
diff changeset
9 # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
edhoprima
parents:
diff changeset
10 # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
edhoprima
parents:
diff changeset
11 # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
edhoprima
parents:
diff changeset
12 # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
edhoprima
parents:
diff changeset
13 # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
edhoprima
parents:
diff changeset
14 # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
edhoprima
parents:
diff changeset
15 # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
16 #
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
17 # Lots of bugs here. Use with care
148
edhoprima
parents:
diff changeset
18 # USE WITH CARE
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
19 #
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
20 # what it does: fetch every picture that has the specified TAGS.
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
21 # requirement: wget, libxslt, openssl
148
edhoprima
parents:
diff changeset
22
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
23 # program additional paths for: cut, sed, wc, openssl, wget, xsltproc, grep
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
24 ADDITIONAL_PATH=
148
edhoprima
parents:
diff changeset
25
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
26 # default server address. Danbooru only! I do not take responsibility of stupidity.
312
110d50856dde moefetch default site updated to yande.re.
Edho Arief <edho@myconan.net>
parents: 305
diff changeset
27 DEFAULT_SITE="yande.re"
148
edhoprima
parents:
diff changeset
28
edhoprima
parents:
diff changeset
29 # base directory. make sure it's writeable. I do not take responsibility if you don't own the folder and files as no check is done for this one.
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
30 # Structure is ${BASE_DIR}/<TAGS>
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
31 # Absolute path only.
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
32 # Leave empty to use whatever folder you're running this at
193
ac6533a8fb51 - Documentation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 192
diff changeset
33 BASE_DIR=
148
edhoprima
parents:
diff changeset
34
edhoprima
parents:
diff changeset
35 # not user modifiable from here
edhoprima
parents:
diff changeset
36
305
21b86001b0c5 [moefetch] Added basic safeguard
Edho Arief <edho@myconan.net>
parents: 302
diff changeset
37 # stop on any error
21b86001b0c5 [moefetch] Added basic safeguard
Edho Arief <edho@myconan.net>
parents: 302
diff changeset
38 set -e
21b86001b0c5 [moefetch] Added basic safeguard
Edho Arief <edho@myconan.net>
parents: 302
diff changeset
39 # ensures all variables initialized
21b86001b0c5 [moefetch] Added basic safeguard
Edho Arief <edho@myconan.net>
parents: 302
diff changeset
40 set -u
302
b90ebadbfd5d Forgot the other wget.
Edho Arief <edho@myconan.net>
parents: 301
diff changeset
41 useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20100101 Firefox/10.0"
b90ebadbfd5d Forgot the other wget.
Edho Arief <edho@myconan.net>
parents: 301
diff changeset
42
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
43 # useless welcome message. Also version
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
44 msg_welcome() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
45 echo "moefetch ${_version}
301
36bc27bb32ff Updated version and copyright.
Edho Arief <edho@myconan.net>
parents: 300
diff changeset
46 Copyright (c) 2009-2012 edogawaconan <edho@myconan.net>
215
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
47 "
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
48 }
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
49
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
50 # Sanitize path. Totally safe. Usage: cmd "$(safe_path "${filename}")"
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
51 safe_path()
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
52 {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
53 # It all depends on the first character.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
54 start=$(printf "%s" "$*" | cut -c 1)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
55 path=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
56 case "${start}" in
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
57 .|/) path="$*";; # . and / is safe. No change.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
58 *) path="./$*";; # Anything else must be prefixed with ./
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
59 esac
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
60 printf "%s" "${path}" # Return.
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
61 }
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
62
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
63 # Checks md5. OpenSSL should be available on anything usable.
230
e922fb1e858f - fixes on openssl output
edhoprima
parents: 229
diff changeset
64 get_md5() { cat "$(safe_path "${1}")" | openssl dgst -md5 | tail -n 1 | sed -e 's/.*\([[:xdigit:]]\{32\}\).*/\1/'; }
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
65
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
66 # Safely get basename.
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
67 get_basename() { basename "$(safe_path "${1}")"; }
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
68
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
69 # Safely get filename (basename without the extension).
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
70 get_filename() { get_basename "${1%.*}"; }
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
71
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
72 # Transformation for tag url.
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
73 get_cleantags() { printf "%s " "$*" | sed -e 's/\&/%26/g;s/=/%3D/g'; }
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
74
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
75 # Returns something if not an md5 value.
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
76 is_not_md5() { get_filename "$1" | sed -e 's/\([0-9a-f]\{32\}\)//g'; }
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
77
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
78
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
79 # fatal error handler
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
80 Err_Fatal() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
81 echo "
214
a6624fb9b317 major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 213
diff changeset
82 Fatal error: ${1}"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
83 exit 1
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
84 }
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
85
215
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
86 Err_Impossible() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
87 echo "
215
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
88 Impossible error. Or you modified content of the working directories when the script is running.
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
89 Please report to moefetch.googlecode.com if you see this message (complete with entire run log)"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
90 exit 1
215
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
91 }
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
92
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
93 # help message
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
94 Err_Help() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
95 echo "moefetch.sh COMMAND [-n] [-p PASSWORD] [-s SITE_URL] [-u USERNAME] TAGS
174
0948e76a57a1 added help. Bump to 0.1-beta2
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 173
diff changeset
96
176
3d2ae9417273 even more improvement
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 175
diff changeset
97 COMMAND:
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
98 (quick)fetch:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
99 Do a complete update. Add prefix quick to skip file checking
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
100 check:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
101 Get list of new files, clean up local folder and print total new files
175
5b7a154dbd21 cosmetics fix for help message
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 174
diff changeset
102
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
103 OPTIONS:
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
104 -n:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
105 Skip checking repository directory.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
106 -p PASSWORD:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
107 Specifies password for login.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
108 -s SITE_URL:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
109 Specify URL of the Danbooru powered site you want to leech from. Default is ${DEFAULT_SITE}.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
110 -u USERNAME:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
111 Specifies username for login.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
112 TAGS:
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
113 Tags you want to download. Separated by spaces. Tag name follows standard Danbooru tagging scheme."
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
114 exit 2
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
115 }
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
116
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
117 # generate link by transforming xml
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
118 Generate_Link() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
119 echo "
215
710082ce6788 major cleanup part2.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 214
diff changeset
120 Fetching XML file"
371
38c7615caf9e Reduced limit to 100.
Edho Arief <edho@myconan.net>
parents: 370
diff changeset
121 pagelimit=100
38c7615caf9e Reduced limit to 100.
Edho Arief <edho@myconan.net>
parents: 370
diff changeset
122 tempnum="${pagelimit}"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
123 iternum=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
124 > "${TEMP_PREFIX}-list"
371
38c7615caf9e Reduced limit to 100.
Edho Arief <edho@myconan.net>
parents: 370
diff changeset
125 while [ "${tempnum}" -ge "${pagelimit}" ]; do
38c7615caf9e Reduced limit to 100.
Edho Arief <edho@myconan.net>
parents: 370
diff changeset
126 url="${SITE}/post/index.xml?tags=$(get_cleantags "${TAGS}")&offset=0&limit=${pagelimit}&page=${iternum}"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
127 [ ${_use_login} -eq 1 ] && url="${url}&login=${LOGIN_USER}&password_hash=${LOGIN_PASS}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
128 wget --no-check-certificate --quiet "${url}" -O "${TEMP_PREFIX}-xml" --referer="${SITE}/post" --user-agent="${useragent}" -e continue=off || Err_Fatal "Failed download catalog file"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
129 printf "Processing XML file... "
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
130 # xslt evilry
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
131 xsltproc - "${TEMP_PREFIX}-xml" <<EOF | sed 's/.*\(https*.*\)\(\/[a-f0-9]\{32\}\).*\.\([^\.]*\)/\1\2.\3/g' | grep ^http > "${TEMP_PREFIX}-templist"
148
edhoprima
parents:
diff changeset
132 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
edhoprima
parents:
diff changeset
133 <xsl:output method="xml" indent="yes"/>
edhoprima
parents:
diff changeset
134 <xsl:template match="post">
edhoprima
parents:
diff changeset
135 <xsl:value-of select="@file_url" />
edhoprima
parents:
diff changeset
136 </xsl:template>
edhoprima
parents:
diff changeset
137 </xsl:stylesheet>
edhoprima
parents:
diff changeset
138 EOF
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
139 tempnum=$(grep -c . "${TEMP_PREFIX}-templist")
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
140 iternum=$((iternum + 1))
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
141 cat "${TEMP_PREFIX}-templist" >> "${TEMP_PREFIX}-list"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
142 echo "${tempnum} file(s) available"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
143 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
144 numfiles=$(grep -c . "${TEMP_PREFIX}-list")
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
145 echo "${numfiles} file(s) available on server"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
146 [ "${numfiles}" -gt 0 ] || Err_Fatal "Error in processing list or no files can be found with specified tag(s) or site."
214
a6624fb9b317 major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 213
diff changeset
147 }
a6624fb9b317 major cleanup. tweaking.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 213
diff changeset
148
148
edhoprima
parents:
diff changeset
149
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
150 progress_init() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
151 _last="-"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
152 printf "${_last}"
205
2e866999c042 now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 204
diff changeset
153 }
2e866999c042 now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 204
diff changeset
154
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
155 progress_anim() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
156 case "${_last}" in
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
157 /) _last="-";;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
158 -) _last=\\;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
159 \\) _last=\|;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
160 \|) _last="/";;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
161 esac
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
162 printf "\b${_last}"
205
2e866999c042 now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 204
diff changeset
163 }
2e866999c042 now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 204
diff changeset
164
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
165 progress_done() { printf "\bdone\n"; }
205
2e866999c042 now with useless animation
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 204
diff changeset
166
200
8efa600ebfdb purge ls
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 199
diff changeset
167 # getting rid of ls (as per suggestion)
8efa600ebfdb purge ls
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 199
diff changeset
168 Count_Files() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
169 numfiles=0
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
170 for dircontent in "${*}/"* "${*}/".*; do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
171 if [ -e "${dircontent}" ] && [ x"${dircontent}" != x"${*}/." ] && [ x"${dircontent}" != x"${*}/.." ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
172 numfiles=$((numfiles + 1))
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
173 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
174 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
175 echo $((numfiles))
200
8efa600ebfdb purge ls
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 199
diff changeset
176 }
8efa600ebfdb purge ls
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 199
diff changeset
177
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
178 # check tools availability
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
179 Check_Tools() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
180 # verify all programs required do indeed exist
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
181 commands="cut sed wc wget xsltproc xargs rm mkdir chown comm grep date openssl"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
182 for cmd in ${commands}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
183 do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
184 [ "$(command -v "${cmd}")" ] || Err_Fatal "${cmd} doesn't exist in ${PATH}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
185 done
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
186 }
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
187
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
188 # verify required folders exist and writeable
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
189 Check_Folders(){
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
190 [ -O "${BASE_DIR}" ] || Err_Fatal "You don't own ${BASE_DIR}. Please fix ${BASE_DIR} or run this script in your own directory."
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
191 for directory in temp trash deleted "${SITE_DIR}/${TARGET_DIR}"; do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
192 if [ ! -d "${BASE_DIR}/${directory}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
193 mkdir -p "${BASE_DIR}/${directory}" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
194 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
195 if [ ! -O "${BASE_DIR}/${directory}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
196 echo "You don't own the ${BASE_DIR}/${directory}, applying globally writeable permission on it"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
197 chmod -R u=rwX,g=rwX,o=rwX "${BASE_DIR}/${directory}" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
198 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
199 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
200 [ "$(Count_Files "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}")" -eq 0 ] && ISNEW=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
201 for i in error ok list newlist templist; do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
202 touch "${TEMP_PREFIX}-${i}" || Fatal_Err "Error creating ${TEMP_PREFIX}-${i}. This shouldn't happen"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
203 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
204 #
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
205 }
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
206
187
efd957294c8c refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 186
diff changeset
207 # Do some cleanup
efd957294c8c refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 186
diff changeset
208 Cleanup_Repository() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
209 # current dir: ${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
210 printf "Cleaning up repository folder... "
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
211 progress_init
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
212 trash_dir="${BASE_DIR}/trash/${trash_dir}/$(date -u "+${SITE_DIR}-${TARGET_DIR}-%Y%m%d-%H.%M")"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
213 trashes="These files have been moved to ${trash_dir}:"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
214 has_trash=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
215 if [ ! -d "${trash_dir}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
216 mkdir -p "${trash_dir}" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
217 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
218 if [ ! -O "${trash_dir}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
219 chmod -R u=rwX,g=rwX,o=rwX "${BASE_DIR}/${directory}" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
220 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
221 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
222 for trash in "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}/"*
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
223 do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
224 if [ -e "${trash}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
225 is_trash=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
226 if [ -d "${trash}" ] || [ -n "$(is_not_md5 "${trash}")" ] || [ -z "$(grep "$(get_basename "${trash}")" "${TEMP_PREFIX}-list")" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
227 is_trash=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
228 has_trash=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
229 mv -f -- "${trash}" "${trash_dir}" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
230 trashes="${trashes}
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
231 $(get_basename "${trash}")"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
232 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
233 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
234 progress_anim
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
235 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
236 rmdir "${trash_dir}" 2>/dev/null
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
237 progress_done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
238 [ -n "${has_trash}" ] && echo "${trashes}"
187
efd957294c8c refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 186
diff changeset
239 }
efd957294c8c refactoring. cleanup. etc.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 186
diff changeset
240
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
241 # check files correctness
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
242 Check_Files() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
243 if [ ! -n "${ISNEW}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
244 [ -z "${NOCLEAN}" ] && Cleanup_Repository
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
245 printf "Checking for errors... "
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
246 progress_init
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
247 files_error="These files do not match its md5:"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
248 files_notdanbooru="These files are not checked:"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
249 has_err_filename=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
250 has_err_md5=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
251 > "${TEMP_PREFIX}-error"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
252 > "${TEMP_PREFIX}-ok"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
253 for file in "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}/"*
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
254 do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
255 if [ -e "${file}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
256 if [ -n "$(is_not_md5 "${file}")" ] || [ -d "${file}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
257 files_notdanbooru="${files_notdanbooru}
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
258 $(get_basename "${file}")"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
259 has_err_filename=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
260 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
261 if [ "$(get_md5 "${file}")" = "$(get_filename "${file}")" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
262 echo "$(get_basename "${file}")" >> "${TEMP_PREFIX}-ok"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
263 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
264 rm "${file}" || Err_Fatal "Error removing ${file}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
265 echo "$(get_basename "${file}")" >> "${TEMP_PREFIX}-error"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
266 files_error="${files_error}
225
265a9ca47a19 - Replaced md5(sum) with openssl. Less platform dependent because the tool is same across platforms
edhoprima
parents: 224
diff changeset
267 $(get_basename "${file}")"
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
268 has_err_md5=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
269 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
270 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
271 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
272 progress_anim
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
273 done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
274 progress_done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
275 if [ ! -n "${has_err_md5}" ] && [ ! -n "${has_err_filename}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
276 echo "All files OK"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
277 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
278 if [ -n "${has_err_md5}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
279 echo "${files_error}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
280 echo "$(grep -c . "${TEMP_PREFIX}-error") file(s) removed"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
281 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
282 [ -n "${has_err_filename}" ] && echo "${files_notdanbooru}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
283 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
284 echo "$(grep -c . "${TEMP_PREFIX}-ok") file(s) available locally"
148
edhoprima
parents:
diff changeset
285
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
286 printf "Generating list of new files... "
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
287 progress_init
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
288 cp -f "${TEMP_PREFIX}-list" "${TEMP_PREFIX}-templist"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
289 while read -r is_ok; do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
290 grep -v "${is_ok}" "${TEMP_PREFIX}-templist" > "${TEMP_PREFIX}-newlist"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
291 cp -f "${TEMP_PREFIX}-newlist" "${TEMP_PREFIX}-templist" || Err_Impossible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
292 progress_anim
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
293 done < "${TEMP_PREFIX}-ok"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
294 progress_done
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
295 echo "$(grep -c . "${TEMP_PREFIX}-newlist") file(s) to be downloaded"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
296 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
297 if [ -n "${ISQUICK}" ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
298 echo "Quick mode selected. Skipping check"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
299 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
300 echo "Empty local repository"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
301 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
302 cat "${TEMP_PREFIX}-list" > "${TEMP_PREFIX}-newlist"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
303 fi
148
edhoprima
parents:
diff changeset
304 }
edhoprima
parents:
diff changeset
305
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
306 # start downloading the images
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
307 Fetch_Images() {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
308 if [ "$(grep -c . "${TEMP_PREFIX}-newlist")" -eq 0 ]; then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
309 echo "No new file"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
310 else
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
311 printf "Downloading files... "
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
312 cd "${BASE_DIR}/${SITE_DIR}/${TARGET_DIR}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
313 wget --no-check-certificate -e continue=on -i "${TEMP_PREFIX}-newlist" -o "${TEMP_PREFIX}.log" --referer="${SITE}/post" --user-agent="${useragent}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
314 fi
148
edhoprima
parents:
diff changeset
315 }
edhoprima
parents:
diff changeset
316
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
317 # initialize base variables and initial command check
227
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
318 init()
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
319 {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
320 # path initialization
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
321 # check if additional path is specified
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
322 if [ -n "${ADDITIONAL_PATH}" ]
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
323 then
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
324 # insert the additional path
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
325 PATH="${ADDITIONAL_PATH}:${PATH}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
326 export PATH
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
327 fi
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
328
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
329 # misc variables
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
330 ISQUICK=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
331 ISNEW=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
332
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
333 # minimum number of arguments: 2 (command and tag). If less than two, exit and print help message
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
334 [ $# -lt 2 ] && Err_Help
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
335 case "$1" in
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
336 check|fetch|quickfetch)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
337 echo "Starting..."
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
338 JOB="$1"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
339 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
340 *)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
341 Err_Help
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
342 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
343 esac
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
344 shift
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
345 SITE=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
346 TAGS=
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
347 has_pass=0
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
348 has_user=0
231
4c0fd276665e - for some reason I broke the getopts logic again. Fixed
edhoprima
parents: 230
diff changeset
349 x=1
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
350 while getopts "s:nu:p:" opt
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
351 do
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
352 case "$opt" in
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
353 s) SITE="$OPTARG";;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
354 n) NOCLEAN=1;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
355 p)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
356 LOGIN_PASS=$(printf "%s" "$OPTARG" | openssl dgst -sha1 | sed -e 's/.*\([[:xdigit:]]\{40\}\).*/\1/')
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
357 has_pass=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
358 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
359 u)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
360 LOGIN_USER="$OPTARG"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
361 has_user=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
362 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
363 esac
231
4c0fd276665e - for some reason I broke the getopts logic again. Fixed
edhoprima
parents: 230
diff changeset
364 x=$OPTIND
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
365 done
231
4c0fd276665e - for some reason I broke the getopts logic again. Fixed
edhoprima
parents: 230
diff changeset
366 shift $(($x-1))
4c0fd276665e - for some reason I broke the getopts logic again. Fixed
edhoprima
parents: 230
diff changeset
367 if [ "$1" = -- ]; then shift; fi
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
368 TAGS="$@"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
369 [ -n "${SITE}" ] || SITE=${DEFAULT_SITE}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
370 [ -n "${TAGS}" ] || Err_Fatal "No tag specified"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
371 # Get base folder - default, current folder or fallback to ${HOME}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
372 [ -n "${BASE_DIR}" ] || BASE_DIR=${PWD}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
373 [ -n "${BASE_DIR}" ] || BASE_DIR=${HOME}
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
374 [ -n "$(echo "${BASE_DIR}" | cut -c1 | grep \/)" ] || BASE_DIR="/${BASE_DIR}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
375 # see if both pass and use are set. If they're set, switch _use_login variable content to 1.
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
376 [ ${has_pass} -eq 1 -a ${has_user} -eq 1 ] && _use_login=1
181
d3b7927bdb2b restructuring and add check if the xml is processed properly
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 180
diff changeset
377
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
378 echo "Tags: ${TAGS}"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
379 # slash is not wanted for folder name
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
380 TARGET_DIR=$(echo "${TAGS}" | sed -e 's/\//_/g')
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
381 SITE_DIR=$(echo "${SITE}" | sed -e 's/\/$//g;s/\//_/g')
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
382 TEMP_PREFIX="${BASE_DIR}/temp/${SITE_DIR}-${TARGET_DIR}"
159
75fe19903b74 Major cleanup
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 158
diff changeset
383 }
148
edhoprima
parents:
diff changeset
384
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
385 # global variables goes here
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
386 init_globals()
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
387 {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
388 _version="1.0-rc3" # version of this script
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
389 _use_login=0 # variable to check whether a login is used or not
228
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
390 }
5d3a0645b504 - Restructured some things.
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 227
diff changeset
391
227
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
392 main()
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
393 {
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
394 # removing GNU-ism as much as possible
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
395 POSIXLY_CORRECT=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
396 #initialize global variables
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
397 init_globals
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
398 #print welcome message
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
399 msg_welcome
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
400 # initialization
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
401 init "$@"
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
402 Check_Tools
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
403 Check_Folders
158
cba73f6a96bb grep check. OpenSolaris' default grep doesn't support -f
edhoprima
parents: 157
diff changeset
404
148
edhoprima
parents:
diff changeset
405
370
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
406 # let's do the job!
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
407 case "${JOB}" in
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
408 check)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
409 Generate_Link
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
410 Check_Files
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
411 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
412 fetch)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
413 Generate_Link
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
414 Check_Files
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
415 Fetch_Images
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
416 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
417 quickfetch)
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
418 ISNEW=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
419 ISQUICK=1
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
420 Generate_Link
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
421 Check_Files
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
422 Fetch_Images
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
423 ;;
fc644e52df7c [bin/ren] Replaced tabs with spaces
Edho Arief <edho@myconan.net>
parents: 324
diff changeset
424 esac
227
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
425 }
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
426
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
427 # call the main routine!
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
428 main "$@"
8b1f6f6b6a3b Bugfixes:
edhoprima@gmail.com <edhoprima@gmail.com>
parents: 226
diff changeset
429