Okay nice another guy didn't reply and ignored a powerdown and he's using it for his peer-to-peer Hentai content delivery network. Definitely an interesting project, they basically built their own BitTorrent for Hentai. Oh god they even have a points system where you can earn Hentai tokens or whatever it's called by thrashing I/O.
Anyway I think that earns our first suspension on TYOC002S (not because I'm anti-henai, it's just abusive I/O 24x7.)
(edit) Once again I want to just mention that I didn't access the customer's service it's just a public site on 443.
All is well with the new AMS IP here as well, though the SolusVM reconfigure networking button broke the network connection because it set a nonexistent network interface name. Debian 11 enumerated the interface as ens3, SolusVM rewrote /etc/network/interfaces as eth0 and then gave a message that the reconfiguration was successful. Easy fix but posting here as a heads-up.
For SMTP, the new IP was blacklisted on https://check.spamhaus.org but their submission tool automatically delisted it without having to wait for human review.
@VirMach - are the new IP address blocks going to get updated in the various geolocation databases? For example Maxmind and IP2location are showing San Mateo, California for the AMS IP. I figure it'll take a few weeks for those to update but checking to see if it's a process that's already been kicked off for the entire address blocks or if users need to submit forms to the different providers manually for each IP address.
@taligentx said: @VirMach - are the new IP address blocks going to get updated in the various geolocation databases?
Yes. It hasn't been kicked off yet we just got them today, we'll start the process after we're done by Friday-ish. And we'll also clean up major blacklistings.
But short answer: no, not true. Currently undetermined what we'll do.
Lmao. Don't you think it's a genius idea that MJJs come up in your stead?
Adding few bucks a month and you get all the Tokyo illegal immigrants, abusers etc. moving their asses out on their own.
Probably more efficient of a deterrent than previous options listed.
Okay I started hearing back from people on storage. I'll be a little vague to maintain privacy.
One guy brought up how the server has gone down (he was one of the people that caused it to lock up once or twice, well one of the main pieces to it happening) in the ticket as if that changes anything in his favor, but he's essentially using it to do video processing. I explained to him that he should consider not using HDD storage for that, so we'll see if he agrees and expands his setup to maybe include an NVMe server to actually do the processing before he stores the large video files on the storage server.
Second guy said he uploads files at one location and downloads files at a second location. That's an interesting way to describe a Plex server that has 100-150 different people connected to it right now. Not that I care, it just means we can't really help him reduce the usage so I hope he's got it figured out.
Third guy said he's using it as a backup server with a script that's not optimized. Strange, he's doing a lot more reads than writes. The said script must be called "qBittorent." But again I hope he figures out how to fix his settings. I would have loved to help him out to reduce the usage but he's not interested either.
By the way I want to re-iterate that I am not in any way accessing the customer's service, just basic tools that show general outputs. Like if an IP is using port 80 then you know they're running a webserver, etc.
Jesus, these could be semi-OK if people just didn't use it as a dedicated CDN, video processing, and streaming site. Even the Plex guy would probably be fine if he wasn't literally mass file sharing on Plex. I guess no one's ever heard of use cases or efficiency.
Does SolusVM not allow you to limit IOPS/CPU per VM?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@AlwaysSkint said:
Does SolusVM not allow you to limit IOPS/CPU per VM?
It does and I was going to consider using it but it's going to pretty much be worthless in most cases since it depends on a lot of other factors. I/O is not something you can really limit properly with just the 3 parameters it allows.
We'd essentially have to put it crazy low and I also haven't researched how it does it, potential bugs, and so on. We don't really mind high reads and writes as long as it's not in tiny 4KB random reads and such. I'll definitely do it on one that seems "appropriate."
Is it Hentai@home? Per my experience, H@H runs fine on a slow HDD with 500Gb of cached content.
Either he was running multiple instances, changing the settings to do more/earn more, or running something alongside it. How much bandwidth does your 500GB setup use? Just so I can evaluate how many multiples of your 500GB. Are you doing the CDN opt-in? Sorry if I sound dumb and it works a different way, I only glanced at it to see if we could find a solution with the customer first but all I saw was minimum I/O requirements, bonus points for it to run 24x7, and when the customer's also ignoring our first message and powerdown and 24x7 at 20-30% utilization rate, then it's a suspension. Not saying he can't appeal.
By the way what's your average transfer size on the disk?
@fan said: H@H should be a no no as the cache contains TONs of small files.
Yeah that's what it looks like on my end looking at the reports.
(edit) Wait so does it work with a big disk and then a smaller disk as cache? Because that would make sense and how people should be setting these up. It's like if you're going to mine Chia (as a bad analogy) it could be fine as long as you're not also doing the I/O intensive portion on the HDD as well.
Will you change my IP again in the future?
I hate to pay for new license for each new IP changes (I know this is not your problem but will be very happy if the IP last like forever?).
Any chance to open Tokyo location for an "honest" and "trustworthy" user like me? hehe.
@Fritz said: Will you change my IP again in the future?
I hate to pay for new license for each new IP changes (I know this is not your problem but will be very happy if the IP last forever?).
If it was up to us, we'd never change it. I was going to actually ask people what they thought about it and realized it would probably be split 50/50 and just make people speculate things but I'll basically do a short version of the post I never made: would people prefer to [a] all keep their IPs and essentially we say yes to anything the IP leasers want to be able to keep it forever which could mean massive price hikes, etc. [b] change IPs and always maintain the same prices and polices, or [c] somewhere in between where whoever wants to keep it has to essentially pay for the price hike plus for everyone else leaving.
Then I realized all the options don't really make sense and a human needs to actually evaluate the situation like we did this time.
@Fritz said:
2. Any chance to open Tokyo location for an "honest" and "trustworthy" user like me? hehe.
If you want to order a plan at current MSRP then I might be able to squeeze you in if you really want it that bad. You'd order the service, look at current price hike for Tokyo, and make a ticket to move there and I'd move you there as soon as possible but your pricing would be "reset" to what the system calculates for Tokyo.
@VirMach said: Wait so does it work with a big disk and then a smaller disk as cache? Because that would make sense and how people should be setting these up. It's like if you're going to mine Chia (as a bad analogy) it could be fine as long as you're not also doing the I/O intensive portion on the HDD as well.
Hentai@home is basically peer-to-peer CDN for e-hentai.org, for the reward token hath. Here's my stats:
546 GB 1,660,272 Files, 3,481 Folders
Bandwidth should be like 100gb/month and disk usage is actually not that much, but the random reading of small files burns the Hitachi HDD on my Kimsufi server pretty hard, making it unable to take other read/write requests. I tried to load that cache on an nvme SSD server and it won't even reach 1% of disk speed, so SSD cache should be very helpful for that kind of usage.
Simple solution: ban H@H for non hybrid VPS. Keep the node workable for others.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
Okay nice another guy didn't reply and ignored a powerdown and he's using it for his peer-to-peer Hentai content delivery network. Definitely an interesting project, they basically built their own BitTorrent for Hentai. Oh god they even have a points system where you can earn Hentai tokens or whatever it's called by thrashing I/O.
Anyway I think that earns our first suspension on TYOC002S (not because I'm anti-henai, it's just abusive I/O 24x7.)
(edit) Once again I want to just mention that I didn't access the customer's service it's just a public site on 443.
I'm shocked that you allow copyright stuffs from Japan hosted on domestic severs. real bold move
Even doujinshi (fanmade) adult manga/douga is the object of investigation (CP regulation and copyright infringements)
Someone reports to the police and your upstream will cut your connection even your contract.
one keyword for you: manga-mura (漫画村) which I believe is not even 100% hentai
If only I knew their sites and plex server hosting these here? real audacious!
Well. It's just that I'm on that server as well and the collateral damage is not what I expected.
You're going to let loose the first three culprits? Sounds like instaban to me.
edit: Just observed CPU steal for a while and it's not any better BTW.
^ Approx. 7% difference. Mein Gott!
Try in another week, or so.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@foitin said: I'm shocked that you allow copyright stuffs from Japan hosted on domestic severs. real bold move
Literally not what was stated at all.
@foitin said: Someone reports to the police and your upstream will cut your connection even your contract.
That's not how legal requests work either.
@foitin said: edit: Just observed CPU steal for a while and it's not any better BTW.
Expected. Read my previous message where I said whatever I said. I'd paraphrase it for you but if you took what I said earlier the way you did it's better you just reference the original message.
@netrix said: just curious, why the new IP has higher ping than the old one @virmach?
Short oversimplified version without checking anything? That'd be difficult and it's still just be a guess. I have a few things in mind it could be especially since it was just recently announced.
I have some ideas for TYOC002S and getting the spikes down but none of them involve suspending abusers. There's some configuration modifications that can be made after I study the traffic patterns further and potential hardware changes which would not happen any time soon (and I'm not sure of a good way to implement it with SolusVM.)
I placed the cancellation request for the $30/y server. It has been offline for a long time since you left CC but I never opened a ticket. I knew the rules of the deal but I was little hopeful
@VirMach said:
Jesus, these could be semi-OK if people just didn't use it as a dedicated CDN, video processing, and streaming site. Even the Plex guy would probably be fine if he wasn't literally mass file sharing on Plex. I guess no one's ever heard of use cases or efficiency.
JFC people are BOLD, makes me feel like my use is a lot more reasonable now.
Expected. Read my previous message where I said whatever I said. I'd paraphrase it for you but if you took what I said earlier the way you did it's better you just reference the original message.
For them to figure out sounds like let them find ways to avoid getting caught to me.
I have some ideas for TYOC002S and getting the spikes down but none of them involve suspending abusers.
Abusers are called abusers for a reason. If you let go of these abusers, you think they'd stop voluntarily?
and what about future storage nodes?
Just by skimming your AUP, the abusers mentioned earlier violated
i. infringes on any third party’s intellectual property or publicity/privacy rights;
ii. violates any applicable law or regulation;
vii. attempts to harm VirMach’s servers or other servers in any way (Attacks, CPU intensive programs for no reason, etc);
xi. involves mass public file storage of any kind (image hosting site, video hosting site, torrents, etc);
xii. involves high CPU/Bandwidth usage, I/O usage, and network usage, or in any other way disrupting the performance of other customers;
xv. involves running a program/script that is designed to use maximum CPU (such as cryptocurrency miners) and not a general fixed amount equal or lesser to the processing power assigned to your service.
and
D) High Usage Policy: Any usage by Customer that disrupts the overall performance of our server(s) is not permitted. Customer agrees to operate within the VirMach usage parameters, which are as follows:
If this isn't enough for suspension I don't know what's qualified then.
It would not happen any time soon
Meanwhile there's no other solution right now.
It's extremely unfair for other legit users too, being taken advantages of.
@foitin said: It's extremely unfair for other legit users..
This. Ban 'em!
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@foitin said: Abusers are called abusers for a reason. If you let go of these abusers, you think they'd stop voluntarily?
and what about future storage nodes?
Read what I previously stated. The abusers are essentially all gone right now, they're not abusing today. The few that were got powered off or suspended. Did it fix the problem? No. Because the abusers aren't the only thing causing what you describe. I've made it very clear in the past that this would most likely be the case.
So okay fine I'll paraphrase my last in-depth message: almost everyone in the Tokyo location is using the service in a manner that causes abrupt and unpredictable spikes, causing the CPU, within a split second, to range anywhere from 15% and max. There is no current solution to this right now outside of randomly suspending everyone until it goes down, because as I mentioned previously, around 90% of people are using Tokyo in this manner, in relatively tiny amounts, but when you add up all of them, it effectively creates this scenario.
If you want to be around people not doing that, request a transfer to Los Angeles or NYC. If you want to remain in Tokyo and believe we should kick off 90% of the people on there, let me know what you're doing on it and private message me your IP so I can determine if you are part of the 90% or the 10% and we'll go from there.
(edit) Getting rid of the abusers may have brought down the OVERALL usage, but it has essentially done NOTHING when it comes to the other issue. So right now it's 15% to max. Previously with abusers it may have been something like 30-40% to max. Getting rid of the abusers will only address the problem where on Saturday the node locks up.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
I have some ideas for TYOC002S and getting the spikes down but none of them involve suspending abusers. There's some configuration modifications that can be made after I study the traffic patterns further and potential hardware changes which would not happen any time soon (and I'm not sure of a good way to implement it with SolusVM.)
I might as well elaborate on these ideas. These are only ideas in my head, I have not yet looked into their realistic possibilities. Parts of it could be not very well thought out, I haven't written it anywhere, planned it anywhere, or anything. Just in my head.
[1] There are some minor tuning we can do when it comes to some RAID settings/configurations where it can skew the performance to where it benefits more in the current environment.
[2] We still have NVMe SSDs on the node and could place larger ones to where it would actually be more effective, and add in some type of caching. I only have really researched LVM caching but SolusVM does not support this automatically which means we'd have to manually configure each virtual server's LVM and then customers would probably have to actually effectively utilize it so I'm throwing that one out the window. I have to see what else we can do without having to rebuild the entire array. On the newer versions of these controllers, they took away SSD caching but it may be possible to do it in some other way with or without the controller. Once again, this is just an idea, it'll require a lot of planning to actually do this, including hardware swaps and configuring the caching. The idea is that caching could potentially handle a lot of the stuff hard drives not meant for and actually reduce the overall spikes.
[3] We still have NVMe SSDs so a second thing we could do here is assign people some NVMe space instead of doing caching. This again requires people effectively utilize it but it might be easier just sending out announcements for this telling people to use the NVMe for certain things. It'll be a much bigger headache actually initially setting it up though and getting people to move certain things over to the NVMe and for certain use cases I'm sure it could be more difficult. The amount of SSDs we have right now is negligible so realistically this would still involve a hardware upgrade.
[4] Board swap. We could try to go with something other than a 5950X with effectively more total power. I assume plenty of people will complain since their core clock would go down. It'd also cost a lot and a lot can go wrong, but this would be a situation where we basically say "let the current weird/extreme usage continue, we'll just add more power." The problem is though that the RAID controller is still the same and I'd have to investigate further, like way way further to ensure that in this case it would actually be a significant improvement so this would possibly be evaluated after at least #1 is done. Because maybe we'd increase the processing power and then instead of 15% to MAX we go to another range but then effectively still have the controller getting bottlenecked and it having the same performance, whether or not some arbitrary number ends up looking better.
almost everyone in the Tokyo location is using the service in a manner that causes abrupt and unpredictable spikes, causing the CPU, within a split second, to range anywhere from 15% and max.
You mean VPN use? Setting up VPNs on storage node doesn't make a lot of sense to me.
If you want to be around people not doing that, request a transfer to Los Angeles or NYC. If you want to remain in Tokyo and believe we should kick off 90% of the people on there, let me know what you're doing on it and private message me your IP so I can determine if you are part of the 90% or the 10% and we'll go from there.
Nah. I'm good. Why LAX (100ms+) or NYC( 160ms+) when TYO has a latency of less than 5ms for me? ISP caps international traffic speed (single thread to US capped at 20Mbps compared to 1Gbps domestic) as well.
My problem is that random inaccessibility on weekend and intermittent irresponsiveness while typing something on SSH.
More likely a CPU issue rather than IO issue.
(edit) Getting rid of the abusers may have brought down the OVERALL usage, but it has essentially done NOTHING when it comes to the other issue. So right now it's 15% to max. Previously with abusers it may have been something like 30-40% to max. Getting rid of the abusers will only address the problem where on Saturday the node locks up.
So you're saying this problem is peculiar to Tokyo storage VPS only and there's no way to solve it?
Is this the reason most Tokyo storage VPSes are not yet provisioned? not until you figure out how to deal with the CPU steal/spike issue?
@foitin said: You mean VPN use? Setting up VPNs on storage node doesn't make a lot of sense to me.
I mean small operation sizes, and in big bursts over a small period of time, and random, but obviously more overall quantity of spikes during peak times. A lot of different use cases can fit into that.
@foitin said: So you're saying this problem is peculiar to Tokyo storage VPS only and there's no way to solve it?
If the "problem" we're speaking of is the part where the CPU goes from 15% to 100% at any given part of a second, which means IO usage is doing that since the IO usage is what's causing the CPU to do that, correct. Outside of maybe the ideas I have to start making a potential impact.
@foitin said: Is this the reason most Tokyo storage VPSes are not yet provisioned? not until you figure out how to deal with the CPU steal/spike issue?
Okay first of all, most Tokyo servers have been provisioned last I checked. So I assume you mean the ones that were in excess of the initial amount promised on that sale. It's of course possible a lot more people ordered since then as they could just force order it and it was also listed as pre-order for some time as well.
But yes this is one small portion of the many reasons on why the second storage node hasn't been sent, because I've been thinking about possibly modifying it before sending it to include one of the hardware related ideas I mentioned.
@codelock said:
LAXA014 has been down for a week now,
More than a week effectively. We know. LAX connectivity issue and other things caused us to not be able to do it when we last wanted to do it (migrating people off.) Now we have these IP changes, other node issues, and we need to get SJCZ005 people off first because that node is in a much worse state so maybe tomorrow at the earliest we can address LAXA014.
(edit) It's technically not down by the way just unusable, just mentioning that as a side note.
@VirMach said:
If you want to be around people not doing that, request a transfer to Los Angeles or NYC. If you want to remain in Tokyo and believe we should kick off 90% of the people on there, let me know what you're doing on it and private message me your IP so I can determine if you are part of the 90% or the 10% and we'll go from there.
I know that wasn't directed at me, but can I take you up on that too? I picked Tokyo because it was the new hotness and I wanted a different physical location than my dedicated server, but now that my new dedi is in San Jose, if I can get better neighbors and ping while freeing up some resources for people that actually need to be in Tokyo, I'd be all over that. I would've just paid for a without-data migration and not bothered anyone about it, but unsurprisingly that's not an option for storage servers.
I mean small operation sizes, and in big bursts over a small period of time, and random, but obviously more overall quantity of spikes during peak times. A lot of different use cases can fit into that.
Speaking of peak time, I'm seeing CPU steal this at 3 AM (which should be similar to most potential abusers?)
So it must have been caused by the non-optimized backup script named qBittorrent?
But yes this is one small portion of the many reasons on why the second storage node hasn't been sent, because I've been thinking about possibly modifying it before sending it to include one of the hardware related ideas I mentioned.
and due to the 10Gbps port delay? Well, 10Gbps might encourage more non-optimized script users I suppose?
Hivelocity is not announcing the IP blocks because apparently announcing anything larger than a /22 (we were announcing a /20 and a /22 in one LOA this time to speed up the process...) kicks in some policy they made up where they have to send an email to the abuse contact for the blocks. They let us know about 14 hours after we provided the revised LOA. A supervisor may or may not look at it and reconsider.
I'm doing the math just in case emergency migrations are required. I have servers I can take to QuadraNet for LAX, but Hivelocity Chicago had all the chunky nodes so only about half max can fit in QuadraNet Chicago. Tampa obviously has no alternative and we already lost one server in QN Miami and a few others in ATL.
QuadraNet and xTom both announced the LOA within 0-2 hours of us sending it in so it shouldn't be a problem re-allocating them outside of getting the IP leasers to reply, but we can always get more blocks off IPXO. And for reference, Dedipath did everything almost immediately, they pointed out the LOA issue, and sent it off to get processed immediately but since they're one step below INAP it's still being processed (but at least it's actually being processed.)
I just found out about this. We're not sending any emergency emails out yet until I have it worked out and a timeline but pretty much unless Hivelocity makes it known that they're willing to be reasonable here we have no other choice outside of several days outage. We could maybe let people decide if I can work out an efficient way to do that. And before we get attacked for not leaving a huge padding on every end in case a provider decided to bring up a policy at the last minute, we've been working on this for 2 weeks and there's been maybe around a 1 day delay (actually I just did the math, it's 15 hours) that can be attributed to us directly, and clearly multiple other providers were able to do the LOA as an LOA is usually done without adding in their own red tape, so it is what it is.
So anyway this is a "potential not smooth sailing" unofficial alert until they make their decision.
(edit) Of course I'm still letting the IP provider know they've done that and to check but they're not exactly 24x7 support either.
Supervisor just replied, it sounds like they're willing to do it.
It's almost shocking how smoothly this went for both my vpses. Part of me expect for things to go terribly wrong and 2 - 3 weeks of downtime as minimum. But nothing like that, both NL and DE servers are up and running with new IPs.
@WeiHo said:
Tokyo Maintenance looks like begin at 09/28.
In fact it is a vase 09/29.
IP changes got pushed back into it. I've still been working on them but not at the speed initially planned. It should still hopefully remain within the 72 hour maintenance window.
the serverstatus page is nicely populated (albeit not in a good way) that it deserves a proper domain like virmachstatus or virmachdownup or haveyoubeenvirmached , so all the real-time datacentre grade literature can be focused on it instead of updating here that may get lost in the pages.
and then here we can get the juicy upcoming plans, expansions, deals, whatdoyouthinks that keep us hyped.
@VirMach said: If the "problem" we're speaking of is the part where the CPU goes from 15% to 100% at any given part of a second, which means IO usage is doing that since the IO usage is what's causing the CPU to do that, correct. Outside of maybe the ideas I have to start making a potential impact.
Is high CPU usage for SATA I/O usual for that particular RAID card?
Shouldn't scattered small reads just stall the requesting VM's virtio thread, as it waits in the controller's io-queue?
@WeiHo said:
Tokyo Maintenance looks like begin at 09/28.
In fact it is a vase 09/29.
IP changes got pushed back into it. I've still been working on them but not at the speed initially planned. It should still hopefully remain within the 72 hour maintenance window.
Comments
Okay nice another guy didn't reply and ignored a powerdown and he's using it for his peer-to-peer Hentai content delivery network. Definitely an interesting project, they basically built their own BitTorrent for Hentai. Oh god they even have a points system where you can earn Hentai tokens or whatever it's called by thrashing I/O.
Anyway I think that earns our first suspension on TYOC002S (not because I'm anti-henai, it's just abusive I/O 24x7.)
(edit) Once again I want to just mention that I didn't access the customer's service it's just a public site on 443.
All is well with the new AMS IP here as well, though the SolusVM reconfigure networking button broke the network connection because it set a nonexistent network interface name. Debian 11 enumerated the interface as ens3, SolusVM rewrote /etc/network/interfaces as eth0 and then gave a message that the reconfiguration was successful. Easy fix but posting here as a heads-up.
For SMTP, the new IP was blacklisted on https://check.spamhaus.org but their submission tool automatically delisted it without having to wait for human review.
@VirMach - are the new IP address blocks going to get updated in the various geolocation databases? For example Maxmind and IP2location are showing San Mateo, California for the AMS IP. I figure it'll take a few weeks for those to update but checking to see if it's a process that's already been kicked off for the entire address blocks or if users need to submit forms to the different providers manually for each IP address.
Overall smooth process, thanks!
Yes. It hasn't been kicked off yet we just got them today, we'll start the process after we're done by Friday-ish. And we'll also clean up major blacklistings.
i heard Plex. oh my waiting for next deal
I bench YABS 24/7/365 unless it's a leap year.
Does SolusVM not allow you to limit IOPS/CPU per VM?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
It does and I was going to consider using it but it's going to pretty much be worthless in most cases since it depends on a lot of other factors. I/O is not something you can really limit properly with just the 3 parameters it allows.
We'd essentially have to put it crazy low and I also haven't researched how it does it, potential bugs, and so on. We don't really mind high reads and writes as long as it's not in tiny 4KB random reads and such. I'll definitely do it on one that seems "appropriate."
Is it Hentai@home? Per my experience, H@H runs fine on a slow HDD with 500Gb of cached content.
Forgot that storage is indeed running on HDD, then H@H should be a no no as the cache contains TONs of small files.
Either he was running multiple instances, changing the settings to do more/earn more, or running something alongside it. How much bandwidth does your 500GB setup use? Just so I can evaluate how many multiples of your 500GB. Are you doing the CDN opt-in? Sorry if I sound dumb and it works a different way, I only glanced at it to see if we could find a solution with the customer first but all I saw was minimum I/O requirements, bonus points for it to run 24x7, and when the customer's also ignoring our first message and powerdown and 24x7 at 20-30% utilization rate, then it's a suspension. Not saying he can't appeal.
By the way what's your average transfer size on the disk?
Yeah that's what it looks like on my end looking at the reports.
(edit) Wait so does it work with a big disk and then a smaller disk as cache? Because that would make sense and how people should be setting these up. It's like if you're going to mine Chia (as a bad analogy) it could be fine as long as you're not also doing the I/O intensive portion on the HDD as well.
@VirMach
So to summarize everything,
Will you change my IP again in the future?
I hate to pay for new license for each new IP changes (I know this is not your problem but will be very happy if the IP last like forever?).
Any chance to open Tokyo location for an "honest" and "trustworthy" user like me? hehe.
https://microlxc.net/
I hate to pay for new license for each new IP changes (I know this is not your problem but will be very happy if the IP last forever?).
If it was up to us, we'd never change it. I was going to actually ask people what they thought about it and realized it would probably be split 50/50 and just make people speculate things but I'll basically do a short version of the post I never made: would people prefer to [a] all keep their IPs and essentially we say yes to anything the IP leasers want to be able to keep it forever which could mean massive price hikes, etc. [b] change IPs and always maintain the same prices and polices, or [c] somewhere in between where whoever wants to keep it has to essentially pay for the price hike plus for everyone else leaving.
Then I realized all the options don't really make sense and a human needs to actually evaluate the situation like we did this time.
If you want to order a plan at current MSRP then I might be able to squeeze you in if you really want it that bad. You'd order the service, look at current price hike for Tokyo, and make a ticket to move there and I'd move you there as soon as possible but your pricing would be "reset" to what the system calculates for Tokyo.
Hentai guy needs to redesign their app.
The push-ups video server has millions of 8KB small objects, yet I received zero suspension from any provider.
We accept Karma donations for the last flan. 🍮 affbrr
@VirMach
I'll wait then. lol.
My LAX IP has changed just 5 minutes ago.
Will wait for GeoIP DB update.
The new IP is on blacklist (Spamhaus ZEN) though, old IP is cleaner.
ISP : Alibaba Inc.
So far everything works as intended.
https://microlxc.net/
Hentai@home is basically peer-to-peer CDN for e-hentai.org, for the reward token hath. Here's my stats:
546 GB 1,660,272 Files, 3,481 Folders
Bandwidth should be like 100gb/month and disk usage is actually not that much, but the random reading of small files burns the Hitachi HDD on my Kimsufi server pretty hard, making it unable to take other read/write requests. I tried to load that cache on an nvme SSD server and it won't even reach 1% of disk speed, so SSD cache should be very helpful for that kind of usage.
Simple solution: ban H@H for non hybrid VPS.
Keep the node workable for others.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I'm shocked that you allow copyright stuffs from Japan hosted on domestic severs. real bold move
Even doujinshi (fanmade) adult manga/douga is the object of investigation (CP regulation and copyright infringements)
Someone reports to the police and your upstream will cut your connection even your contract.
one keyword for you: manga-mura (漫画村) which I believe is not even 100% hentai
If only I knew their sites and plex server
hosting these here? real audacious!
Well. It's just that I'm on that server as well and the collateral damage is not what I expected.
You're going to let loose the first three culprits? Sounds like instaban to me.
edit: Just observed CPU steal for a while and it's not any better BTW.
just curious, why the new IP has higher ping than the old one @virmach?
ping from SG
^ Approx. 7% difference.
Mein Gott!
Try in another week, or so.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Literally not what was stated at all.
That's not how legal requests work either.
Expected. Read my previous message where I said whatever I said. I'd paraphrase it for you but if you took what I said earlier the way you did it's better you just reference the original message.
Short oversimplified version without checking anything? That'd be difficult and it's still just be a guess. I have a few things in mind it could be especially since it was just recently announced.
Dedi's gone
Just when I found something useful for it
I have some ideas for TYOC002S and getting the spikes down but none of them involve suspending abusers. There's some configuration modifications that can be made after I study the traffic patterns further and potential hardware changes which would not happen any time soon (and I'm not sure of a good way to implement it with SolusVM.)
The new ones or old ones?
I placed the cancellation request for the $30/y server. It has been offline for a long time since you left CC but I never opened a ticket. I knew the rules of the deal but I was little hopeful
JFC people are BOLD, makes me feel like my use is a lot more reasonable now.
For them to figure out sounds like let them find ways to avoid getting caught to me.
Abusers are called abusers for a reason. If you let go of these abusers, you think they'd stop voluntarily?
and what about future storage nodes?
Just by skimming your AUP, the abusers mentioned earlier violated
and
D) High Usage Policy: Any usage by Customer that disrupts the overall performance of our server(s) is not permitted. Customer agrees to operate within the VirMach usage parameters, which are as follows:If this isn't enough for suspension I don't know what's qualified then.
Meanwhile there's no other solution right now.
It's extremely unfair for other legit users too, being taken advantages of.
This. Ban 'em!
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Read what I previously stated. The abusers are essentially all gone right now, they're not abusing today. The few that were got powered off or suspended. Did it fix the problem? No. Because the abusers aren't the only thing causing what you describe. I've made it very clear in the past that this would most likely be the case.
So okay fine I'll paraphrase my last in-depth message: almost everyone in the Tokyo location is using the service in a manner that causes abrupt and unpredictable spikes, causing the CPU, within a split second, to range anywhere from 15% and max. There is no current solution to this right now outside of randomly suspending everyone until it goes down, because as I mentioned previously, around 90% of people are using Tokyo in this manner, in relatively tiny amounts, but when you add up all of them, it effectively creates this scenario.
If you want to be around people not doing that, request a transfer to Los Angeles or NYC. If you want to remain in Tokyo and believe we should kick off 90% of the people on there, let me know what you're doing on it and private message me your IP so I can determine if you are part of the 90% or the 10% and we'll go from there.
(edit) Getting rid of the abusers may have brought down the OVERALL usage, but it has essentially done NOTHING when it comes to the other issue. So right now it's 15% to max. Previously with abusers it may have been something like 30-40% to max. Getting rid of the abusers will only address the problem where on Saturday the node locks up.
@VirMach My VPS has been offline for a few days. :'(
Well...
What is wrong with this people? seriously
^ Wow!
(Directed at Virmach's post.)
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I might as well elaborate on these ideas. These are only ideas in my head, I have not yet looked into their realistic possibilities. Parts of it could be not very well thought out, I haven't written it anywhere, planned it anywhere, or anything. Just in my head.
[1] There are some minor tuning we can do when it comes to some RAID settings/configurations where it can skew the performance to where it benefits more in the current environment.
[2] We still have NVMe SSDs on the node and could place larger ones to where it would actually be more effective, and add in some type of caching. I only have really researched LVM caching but SolusVM does not support this automatically which means we'd have to manually configure each virtual server's LVM and then customers would probably have to actually effectively utilize it so I'm throwing that one out the window. I have to see what else we can do without having to rebuild the entire array. On the newer versions of these controllers, they took away SSD caching but it may be possible to do it in some other way with or without the controller. Once again, this is just an idea, it'll require a lot of planning to actually do this, including hardware swaps and configuring the caching. The idea is that caching could potentially handle a lot of the stuff hard drives not meant for and actually reduce the overall spikes.
[3] We still have NVMe SSDs so a second thing we could do here is assign people some NVMe space instead of doing caching. This again requires people effectively utilize it but it might be easier just sending out announcements for this telling people to use the NVMe for certain things. It'll be a much bigger headache actually initially setting it up though and getting people to move certain things over to the NVMe and for certain use cases I'm sure it could be more difficult. The amount of SSDs we have right now is negligible so realistically this would still involve a hardware upgrade.
[4] Board swap. We could try to go with something other than a 5950X with effectively more total power. I assume plenty of people will complain since their core clock would go down. It'd also cost a lot and a lot can go wrong, but this would be a situation where we basically say "let the current weird/extreme usage continue, we'll just add more power." The problem is though that the RAID controller is still the same and I'd have to investigate further, like way way further to ensure that in this case it would actually be a significant improvement so this would possibly be evaluated after at least #1 is done. Because maybe we'd increase the processing power and then instead of 15% to MAX we go to another range but then effectively still have the controller getting bottlenecked and it having the same performance, whether or not some arbitrary number ends up looking better.
You mean VPN use? Setting up VPNs on storage node doesn't make a lot of sense to me.
Nah. I'm good. Why LAX (100ms+) or NYC( 160ms+) when TYO has a latency of less than 5ms for me? ISP caps international traffic speed (single thread to US capped at 20Mbps compared to 1Gbps domestic) as well.
My problem is that random inaccessibility on weekend and intermittent irresponsiveness while typing something on SSH.
More likely a CPU issue rather than IO issue.
So you're saying this problem is peculiar to Tokyo storage VPS only and there's no way to solve it?
Is this the reason most Tokyo storage VPSes are not yet provisioned? not until you figure out how to deal with the CPU steal/spike issue?
I just requested this since originally my VPS was in NY but somehow got ryzen migrated to Tokyo
The service status says TYOC040 (solved). But my TYOC040 vps is still offline. It's been a month. Do I need to submit a work order?
Work orders can be submitted on OGF.
I mean small operation sizes, and in big bursts over a small period of time, and random, but obviously more overall quantity of spikes during peak times. A lot of different use cases can fit into that.
If the "problem" we're speaking of is the part where the CPU goes from 15% to 100% at any given part of a second, which means IO usage is doing that since the IO usage is what's causing the CPU to do that, correct. Outside of maybe the ideas I have to start making a potential impact.
Okay first of all, most Tokyo servers have been provisioned last I checked. So I assume you mean the ones that were in excess of the initial amount promised on that sale. It's of course possible a lot more people ordered since then as they could just force order it and it was also listed as pre-order for some time as well.
But yes this is one small portion of the many reasons on why the second storage node hasn't been sent, because I've been thinking about possibly modifying it before sending it to include one of the hardware related ideas I mentioned.
LAXA014 has been down for a week now,
Want free vps ? https://microlxc.net
More than a week effectively. We know. LAX connectivity issue and other things caused us to not be able to do it when we last wanted to do it (migrating people off.) Now we have these IP changes, other node issues, and we need to get SJCZ005 people off first because that node is in a much worse state so maybe tomorrow at the earliest we can address LAXA014.
(edit) It's technically not down by the way just unusable, just mentioning that as a side note.
I know that wasn't directed at me, but can I take you up on that too? I picked Tokyo because it was the new hotness and I wanted a different physical location than my dedicated server, but now that my new dedi is in San Jose, if I can get better neighbors and ping while freeing up some resources for people that actually need to be in Tokyo, I'd be all over that. I would've just paid for a without-data migration and not bothered anyone about it, but unsurprisingly that's not an option for storage servers.
Speaking of peak time, I'm seeing CPU steal this at 3 AM (which should be similar to most potential abusers?)
So it must have been caused by the non-optimized backup script named qBittorrent?
and due to the 10Gbps port delay? Well, 10Gbps might encourage more non-optimized script users I suppose?
Virmach el diablo.
Hivelocity is not announcing the IP blocks because apparently announcing anything larger than a /22 (we were announcing a /20 and a /22 in one LOA this time to speed up the process...) kicks in some policy they made up where they have to send an email to the abuse contact for the blocks. They let us know about 14 hours after we provided the revised LOA. A supervisor may or may not look at it and reconsider.
I'm doing the math just in case emergency migrations are required. I have servers I can take to QuadraNet for LAX, but Hivelocity Chicago had all the chunky nodes so only about half max can fit in QuadraNet Chicago. Tampa obviously has no alternative and we already lost one server in QN Miami and a few others in ATL.
QuadraNet and xTom both announced the LOA within 0-2 hours of us sending it in so it shouldn't be a problem re-allocating them outside of getting the IP leasers to reply, but we can always get more blocks off IPXO. And for reference, Dedipath did everything almost immediately, they pointed out the LOA issue, and sent it off to get processed immediately but since they're one step below INAP it's still being processed (but at least it's actually being processed.)
I just found out about this. We're not sending any emergency emails out yet until I have it worked out and a timeline but pretty much unless Hivelocity makes it known that they're willing to be reasonable here we have no other choice outside of several days outage. We could maybe let people decide if I can work out an efficient way to do that. And before we get attacked for not leaving a huge padding on every end in case a provider decided to bring up a policy at the last minute, we've been working on this for 2 weeks and there's been maybe around a 1 day delay (actually I just did the math, it's 15 hours) that can be attributed to us directly, and clearly multiple other providers were able to do the LOA as an LOA is usually done without adding in their own red tape, so it is what it is.
So anyway this is a "potential not smooth sailing" unofficial alert until they make their decision.
(edit) Of course I'm still letting the IP provider know they've done that and to check but they're not exactly 24x7 support either.
Supervisor just replied, it sounds like they're willing to do it.
It's almost shocking how smoothly this went for both my vpses. Part of me expect for things to go terribly wrong and 2 - 3 weeks of downtime as minimum. But nothing like that, both NL and DE servers are up and running with new IPs.
@VirMach, is NYCB033X done? can I use new i.p?
Tokyo Maintenance looks like begin at 09/28.
In fact it is a vase 09/29.
https://billing.virmach.com/serverstatus.php
Not yet.
IP changes got pushed back into it. I've still been working on them but not at the speed initially planned. It should still hopefully remain within the 72 hour maintenance window.
the serverstatus page is nicely populated (albeit not in a good way) that it deserves a proper domain like virmachstatus or virmachdownup or haveyoubeenvirmached , so all the real-time datacentre grade literature can be focused on it instead of updating here that may get lost in the pages.
and then here we can get the juicy upcoming plans, expansions, deals, whatdoyouthinks that keep us hyped.
I bench YABS 24/7/365 unless it's a leap year.
Is high CPU usage for SATA I/O usual for that particular RAID card?
Shouldn't scattered small reads just stall the requesting VM's virtio thread, as it waits in the controller's io-queue?
You look optimistic.