root/branches/feature-server/plagger/lib/Plagger/Plugin/CustomFeed/MixiDiarySearch.pm

Revision 1009 (checked in by miyagawa, 3 years ago)

r2680@rock (orig r938): miyagawa | 2006-06-09 10:28:17 +0900
add podtrac TruePermalink?. via http://d.hatena.ne.jp/mryfmo/20060608
r2681@rock (orig r939): miyagawa | 2006-06-09 10:30:48 +0900
TruePermalink?: add feedburner podcast redirector. Refs #226
r2682@rock (orig r940): miyagawa | 2006-06-09 16:11:35 +0900
use Last-Modified header to populate entry date, even if handler can't find one.
via http://subtech.g.hatena.ne.jp/otsune/20060608/norkdailymemo
r2683@rock (orig r941): miyagawa | 2006-06-09 16:12:52 +0900
take off utf-8 flag when taking digest value
r2684@rock (orig r942): miyagawa | 2006-06-09 17:04:38 +0900

Publish
CHTML: Don't die if body contains non-sjis mappable characters
r2685@rock (orig r943): miyagawa | 2006-06-09 17:26:01 +0900
defaults to cp932 would be better
r2686@rock (orig r944): miyagawa | 2006-06-09 17:37:37 +0900

r2687@rock (orig r945): miyagawa | 2006-06-09 18:48:15 +0900
add pya.cc upgrader via http://subtech.g.hatena.ne.jp/otsune/20060608/pya2feed
r2688@rock (orig r946): miyagawa | 2006-06-09 21:21:47 +0900
CustomFeed?
2chSearch
r2689@rock (orig r947): miyagawa | 2006-06-09 21:26:31 +0900
oops, remove </b>
r2690@rock (orig r948): miyagawa | 2006-06-09 21:44:42 +0900
fix date if it found true entry
r2691@rock (orig r949): miyagawa | 2006-06-09 21:59:05 +0900
need quotes
r2692@rock (orig r950): miyagawa | 2006-06-09 22:06:35 +0900
Planet: Scrubber support back inlib/Plagger/Plugin/Publish/Planet.pm
r2693@rock (orig r951): miyagawa | 2006-06-09 22:08:01 +0900
oops
r2694@rock (orig r952): otsune | 2006-06-09 22:11:04 +0900
fix extract http://pyc.cc/

r2695@rock (orig r953): otsune | 2006-06-09 22:12:28 +0900
add EntryFulltext? for seesaa blog

r2696@rock (orig r954): otsune | 2006-06-09 23:27:11 +0900
fix %3A

r2697@rock (orig r955): miyagawa | 2006-06-10 02:26:28 +0900
MixiDiarySearch?: decode keyword query
r2698@rock (orig r956): miyagawa | 2006-06-10 02:53:41 +0900
TruePermalink? enbug stuff. Use permlalink to find handlers
r2699@rock (orig r957): otsune | 2006-06-10 03:08:33 +0900
add EntryFulltext? http://headlines.yahoo.co.jp/

r2700@rock (orig r958): otsune | 2006-06-10 04:38:27 +0900
add Apple KB and TIL document

r2701@rock (orig r959): otsune | 2006-06-10 04:43:22 +0900
oops.

r2702@rock (orig r960): miyagawa | 2006-06-10 23:07:48 +0900
set Bloglines n=100
r2703@rock (orig r961): miyagawa | 2006-06-11 01:35:38 +0900
MixiDiarySearch?: allow no_photo.gif
r2704@rock (orig r962): miyagawa | 2006-06-11 01:45:53 +0900
2chSearh: Fix error handling
r2705@rock (orig r963): miyagawa | 2006-06-11 02:07:11 +0900
added takesako-san for his patch
r2706@rock (orig r964): otsune | 2006-06-11 05:59:58 +0900
modified Chugoku SHinbun, add EFT for http://www.zianplus.net/

r2707@rock (orig r965): otsune | 2006-06-11 10:17:02 +0900
add pMachine ExpressionEngine? http://www.pmachine.com/

r2708@rock (orig r966): youpy | 2006-06-11 12:38:21 +0900
fix regexp

r2709@rock (orig r967): otsune | 2006-06-12 04:09:24 +0900
fix extract regexp

r2710@rock (orig r968): otsune | 2006-06-12 04:13:19 +0900
update regexp

r2711@rock (orig r969): otsune | 2006-06-12 04:29:18 +0900
support http://www.mainichi-msn.co.jp/photo/etc/photo_feature/

r2712@rock (orig r970): otsune | 2006-06-12 06:08:15 +0900
fix wordpress.
Add mainichi-msn Photo and separate handle.
Add http://www.actiblog.com/

r2713@rock (orig r971): otsune | 2006-06-12 07:02:23 +0900
refine livedoorblog.pl
fix miss.

r2714@rock (orig r972): miyagawa | 2006-06-12 13:25:28 +0900
extract_title should be case insensitive. via http://d.hatena.ne.jp/sfujiwara/20060611/1150051152
r2715@rock (orig r973): miyagawa | 2006-06-12 13:39:12 +0900
rewrite config doesn't die even if it can't rewrite because of permission problem
r2716@rock (orig r974): miyagawa | 2006-06-12 13:43:25 +0900
skip all livedoorkeyword link
r2719@rock (orig r975): otsune | 2006-06-12 14:50:19 +0900
fix misc regexp

r2720@rock (orig r976): miyagawa | 2006-06-12 15:44:57 +0900
support handle only in livedoorblog.pl to work with aggregated feeds
r2721@rock (orig r977): miyagawa | 2006-06-12 18:22:40 +0900
TruePermalink? for blogpeople redirector
r2722@rock (orig r978): otsune | 2006-06-12 22:14:03 +0900
opps 'Unmatched ( in regex;'

r2723@rock (orig r979): youpy | 2006-06-13 10:21:42 +0900
add mailman upgrader


r2724@rock (orig r980): youpy | 2006-06-13 10:28:19 +0900
fix handle regexp


r2727@rock (orig r983): miyagawa | 2006-06-13 19:00:22 +0900
Subscription
Planet: add feedster.jp
r2728@rock (orig r984): miyagawa | 2006-06-13 19:06:06 +0900
use lang/all on feedster.jp
r2734@rock (orig r985): otsune | 2006-06-13 22:11:21 +0900
fix regexp

r2735@rock (orig r986): miyagawa | 2006-06-14 00:34:01 +0900
new plugin Notify
Beep
r2736@rock (orig r987): miyagawa | 2006-06-14 00:34:40 +0900
planet: remove unnecessary bit
r2737@rock (orig r988): miyagawa | 2006-06-14 00:35:03 +0900
update example to use sixapart-std
r2738@rock (orig r989): otsune | 2006-06-14 02:55:47 +0900
remove icon_re. RecentComment? can't get it

r2745@rock (orig r990): miyagawa | 2006-06-14 12:07:29 +0900
t/core is for developer test and not needed for installers
r2746@rock (orig r991): miyagawa | 2006-06-14 12:49:00 +0900
support mixi_tos_paranoia mode
r2747@rock (orig r992): miyagawa | 2006-06-14 13:10:40 +0900
title would be ok
r2792@rock (orig r993): miyagawa | 2006-06-16 15:04:12 +0900
  • New plugin Subscription::Bookmarks (and its IE subclass) to read IE favorites.
r2793@rock (orig r994): miyagawa | 2006-06-16 15:11:52 +0900
added TODO as comment
r2794@rock (orig r995): youpy | 2006-06-17 20:36:18 +0900
add Plugin::Subscription::Bookmarks
Safari


r2795@rock (orig r996): youpy | 2006-06-17 21:39:18 +0900
add tag support by folder name


r2796@rock (orig r997): youpy | 2006-06-18 15:41:59 +0900
use $uri->file when scheme is 'file'


r2797@rock (orig r998): youpy | 2006-06-18 15:42:56 +0900
add Plugin::Subscription::Bookmarks
Mozilla


r2798@rock (orig r999): miyagawa | 2006-06-19 15:23:13 +0900
bump URI
Fetch req
r2800@rock (orig r1000): miyagawa | 2006-06-22 00:26:46 +0900
dependency for Bookmarks
Safari. 1000th commit!
r2801@rock (orig r1001): miyagawa | 2006-06-22 00:30:57 +0900
fix config rewriting bug when the password contains regexp metachars. via http://d.hatena.ne.jp/sfujiwara/20060621/1150899012
r2802@rock (orig r1002): otsune | 2006-06-22 00:54:24 +0900
add http://www.computerworld.jp/ http://autopage.teacup.com/
fix headlines_yahoo_jp (Thanks woremacx)
fix goo blog

r2803@rock (orig r1003): miyagawa | 2006-06-22 01:10:00 +0900
import drawnboy's EntryFullText? yamls via http://svn.nowherenear.net/repos/public/misc/eft/
r2804@rock (orig r1004): miyagawa | 2006-06-22 01:10:39 +0900
update AUTHOR
r2805@rock (orig r1005): s_nobu | 2006-06-22 06:17:15 +0900
require HTML
Entities for enclosure support.

r2807@rock (orig r1006): miyagawa | 2006-06-22 15:46:30 +0900
URI
Fetch 0.07 is broken (i was a moron), reverting back to 0.06 for now
r2808@rock (orig r1007): miyagawa | 2006-06-22 16:04:48 +0900
packaging 0.7.3
Line 
1 package Plagger::Plugin::CustomFeed::MixiDiarySearch;
2 use strict;
3 use base qw( Plagger::Plugin );
4
5 use Encode;
6 use HTML::Entities;
7 use Plagger::UserAgent;
8 use Plagger::Util qw( decode_content );
9
10 sub register {
11     my($self, $context) = @_;
12     $context->register_hook(
13         $self,
14         'customfeed.handle' => \&handle,
15     );
16 }
17
18 sub handle {
19     my($self, $context, $args) = @_;
20
21     if ($args->{feed}->url =~ m!^http://mixi\.jp/search_diary\.pl\?.*keyword=!) {
22         $self->aggregate($context, $args);
23         return 1;
24     }
25
26     return;
27 }
28
29 sub aggregate {
30     my($self, $context, $args) = @_;
31
32     my $url = $args->{feed}->url;
33     $context->log(info => "GET $url");
34
35     my $agent = Plagger::UserAgent->new;
36     my $res = $agent->fetch($url, $self);
37
38     if ($res->is_error) {
39         $context->log(error => "GET $url failed: " . $res->status_code);
40         return;
41     }
42
43     my $content = decode_content($res);
44
45     my %query = URI->new($url)->query_form;
46
47     # heh, this is a "Cache"
48     my $title = "mixi: Search for " . decode("euc-jp", $query{keyword});
49     if (!$self->conf->{mixi_tos_paranoia}) {
50         $title .= " (Cache)";
51     }
52
53     my $feed = $args->{feed};
54     $feed->title($title);
55     $feed->link($url);
56
57     my $re = decode('utf-8', <<'RE');
58 <table BORDER=0 CELLSPACING=1 CELLPADDING=4 WIDTH=550>
59 <tr>
60 <td WIDTH=90 VALIGN=top ROWSPAN=5 ALIGN=center background=http://img\.mixi\.jp/img/bg_line\.gif><a href="(view_diary\.pl\?id=\d+&owner_id=\d+)"><img SRC="(http://img\.mixi\.jp/photo/member/.*?\.\w+)" VSPACE=3 border=0></a></td>
61 <td BGCOLOR=#FDF9F2><font COLOR=#996600>名&nbsp;&nbsp;前</font></td>
62 <td COLSPAN=2 BGCOLOR=#FFFFFF>(.*?) \(.*?\)
63
64 </td></tr>
65
66 <tr>
67 <td BGCOLOR=#FDF9F2><font COLOR=#996600>タイトル</font></td>
68 <td COLSPAN=2 BGCOLOR=#FFFFFF>(.*?)</td></tr>
69
70 <tr>
71 <td BGCOLOR=#FDF9F2><font COLOR=#996600>本&nbsp;&nbsp;文</font></td>
72 <td COLSPAN=2 BGCOLOR=#FFFFFF>(.*?)</td></tr>
73
74
75 <tr>
76 <td NOWRAP BGCOLOR=#FDF9F2 WIDTH=80><font COLOR=#996600>作成日時</font></td>
77 <td BGCOLOR=#FFFFFF WIDTH=220>(\d\d月\d\d日 \d\d:\d\d)</td>
78 RE
79
80     $content =~ s/\r\n/\n/g;
81
82     my @matches;
83     my @keys = qw( link photo name title body date );
84     my $date_format = decode("utf-8", "%Y %m月%d日 %H:%M");
85
86     while ($content =~ /$re/gs) {
87         my $data;
88         @{$data}{@keys} = ($1, $2, $3, $4, $5, $6);
89
90         my $now = Plagger::Date->now;
91         my $current = $now->year;
92         $data->{date} = Plagger::Date->strptime($date_format, "$current $data->{date}");
93
94         $data->{date}->set_time_zone('Asia/Tokyo'); # set floating datetime
95
96         # one year ago, if the parsed datetime is in the future
97         if ($data->{date} > $now) {
98             $data->{date}->subtract(years => 1);
99         }
100
101         $data->{date}->set_time_zone(Plagger->context->conf->{timezone} || 'local');
102
103         my $entry = Plagger::Entry->new;
104
105         $entry->title($data->{title});
106         $entry->link( URI->new_abs($data->{link}, $url) );
107         $entry->date($data->{date});
108
109         unless ($self->conf->{mixi_tos_paranoia}) {
110             $entry->body( munge_body($data->{body}) );
111             $entry->icon({ url => URI->new_abs($data->{photo}, $url) });
112             $entry->author( decode_entities($data->{name}) );
113         }
114
115         $feed->add_entry($entry);
116     }
117
118     $context->update->add($feed);
119 }
120
121 sub munge_body {
122     my $body = shift;
123     $body =~ s/<br>//g;
124     decode_entities($body);
125 }
126
127 1;
128
129 __END__
130
131 =head1 NAME
132
133 Plagger::Plugin::CustomFeed::MixiDiarySearch - Custom feed for mixi diary search
134
135 =head1 SYNOPSIS
136
137   global:
138     user_agent:
139       cookies: /path/to/cookies.txt
140
141   plugins:
142     - module: Subscription::Config
143       config:
144         feed:
145           - http://mixi.jp/search_diary.pl?submit=search&keyword=Plagger
146     - module: CustomFeed::MixiDiarySearch
147
148 =head1 DESCRIPTION
149
150 This plugin creates a custom feed off of Mixi diary search. Since mixi
151 requires login authentication for all pages, this plugin also requires
152 a valid login cookie set to global I<user_agent> config.
153
154 =head1 AUTHOR
155
156 Tatsuhiko Miyagawa
157
158 =head1 SEE ALSO
159
160 L<Plagger>, L<http://mixi.jp/>
161
162 =cut
Note: See TracBrowser for help on using the browser.